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Preface 


For the forty second time, the Association for Educational Communications and Technology (AECT) is 
sponsoring the publication of these Proceedings. Papers published in this volume were presented at the 
annual AECT Convention in Las Vegas, NV. A limited quantity of these Proceedings were printed and 
sold in both hardcopy and electronic versions. Volumes | and 2 are available through the Educational 
Resources Clearinghouse (ERIC) System. Proceedings volumes are available to members at AECT.ORG. 
Proceedings copies are also available at: 


http://www.tresystems.com/proceedings/ 


The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. This year, both volumes 
are included in one document. 


REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 


Michael R. Simonson 
Deborah J. Seepersaud 
Editors 
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Progress monitoring is used to assess a student’s performance during the early stages of literacy 
development. Computerized progress monitoring systems are capable of scoring some progress monitoring 
measures automatically. However, other measures, such as those involving writing or sign language, are 
typically scored manually which is time consuming and potentially less reliable than computer scoring. In this 
project, we designed DeepSign, a Deep Learning software tool, to auto-score a sign-language based picture- 
naming measure. DeepSign generates immediate feedback that can enhance progress monitoring and reduce 
teacher workload. 


Introduction 


Progress Monitoring (PM) is used to measure a student’s level of academic performance and to 
determine whether students are making adequate academic progress (e.g., McMaster et al., 2017). In recent 
years, several computerized PM systems have been developed (e.g. AIMSweb (Shinn & Shinn, 2002), 
FastBridge (FastBridge, 2019), DIBELS, etc.) and research on PM has increased (c.f. Ysseldyke & Tardrew, 
2007). 
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However, many web-based systems are invalid for students with special needs such as those who are 
Deaf or Hard of Hearing (DHH) as computerized PM systems are often auditory based. To address this issue, 
we developed AvePM, a PM software suite designed with an emphasis on students who are DHH (Hooper, 
Rose, & Sperling, 2018). AvePM uses curriculum-based measurement (CBM), which includes measures that are 
brief, easy to complete, and produce reliable and valid data that can be used to improve academic outcomes 
(Deno, 1985; Hosp et al., 2016). CBM has been used successfully in education for many years (Miura et al., 
2007; Shinn, 1989). 


Progress Monitoring and Deep Learning 


In this paper, we describe our attempts to automate scoring of an app that is part of the AvePM 
software suite. The suite (http://avepm.com) includes eight measures of student performance addressing tasks 
such as emergent and early reading, ASL and English development, as well as more-traditional reading and 
writing measures. Of the eight measures, two are scored by computer algorithms, but the other six are scored 
manually using custom tools developed for each measure. 

One of the measures in the AvePM suite is named PictureNaming. Picture naming is a commonly used 
progress-monitoring task that measures students’ ability to name pictures—an important first step in the reading 
process. PictureNaming was designed to accommodate students who are DHH. The app presents randomly 
selected images and students sign or speak the image names; a video recording is stored online and made 
available for manual scoring. The tool takes video input thereby allowing students to record responses in sign 
language. 

Although some progress monitoring tasks (e.g. cloze tests) can be scored reliably by a computer (Fuchs 
et al., 1984; Miller et al., 2008), to date, tasks recorded using the PictureNaming app have been scored manually 
rather than by machines due to the complexity of scoring ASL. In the current project, we developed a Deep 
Learning algorithm to auto-score Picture Naming. Our goal was to provide immediate feedback for students and 
to transform the scoring and subsequent data analysis processes for teachers. 


Deep Learning for Video recognition and Classification 

Deep Learning, a subset of machine learning in artificial intelligence, is used for image classification, 
object detection and face recognition. The term ‘Deep’ in Deep Learning refers to the number of layers through 
which the data are transformed. These layers extract high-level features from the raw input. Features are the 
variables or attributes of a dataset such as grayscale values of pixels, edges, objects, and points. For example, 
when classifying a person as a male or a female, the features might include height, weight, body-mass index, 
hair length, speech pitch, etc. 

The term neural network is used widely in Deep Learning. Neural networks are computing systems 
designed for pattern recognition. A neural network is a collection of connected units known as neurons. 
Computer neurons are similar to biological neurons: connections between neurons are like synapses that 
transmit signal between neurons in the brain. There are many types of neural networks. Each functions 
according the task it must perform such as text, speech, or image processing. 

A neural network consists of many layers. The first layer, the input layer, takes features of an object as 
input; the last layer, the output layer, classifies, or recognizes, the object. All other layers (known as hidden 
layers) processes the input data (which is in the form of features) to produce some form of predication. Recent 
developments in Deep Learning have greatly improved the accuracy and precision of visual object recognition 
(LeCun, Bengio, & Hinton, 2015). The current project applies Deep Learning to ASL recognition. 

Advancements in Deep Learning technology and the availability of computational power have resulted 
in considerable interest in video and image recognition. For example, algorithms have been developed to train 
machines to differentiate between cats and dogs (Branson, et al., 2014), path tracking for automated self-driving 
cars (Paden, et al., 2016), and so forth. In each case, an effective recognition model is dependent on the 
generation of a dataset that is aligned with the problem being solved. 

However, little research has been conducted on sign language recognition (Zafrulla, et al., 2011, Chai, 
et al., 2013), thus datasets needs to be constructed to build an effective recognition model. Prior work has used 
the Microsoft Kinect sensor for recording videos that segregate features such as hand color, shape and size, and 
uses pixel values to track the body movement (Ren et al. 2013; Li et al., 2012,). Prior research has been limited 
by a scarcity of data, difficulties associated with identifying human features (e.g. differences in hand shape, size 
and color), and limitations associated with the development of an effective recognition algorithm. 

Our short-term goal was to develop a Deep Learning algorithm that is capable of automatically scoring 
video recordings and generating useful feedback. Our long-term goal is to integrate the algorithm into the 
PictureNaming app, to improve scoring efficiency and reliability and reduce teacher-scoring time. 
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PictureNaming Tools and Interfaces 

The picture-naming tool currently used within the AvePM system includes three components: A 
student interface that captures videos of students signing the names of images; a manual-scoring tool to view 
and score the students’ videos; and a data chart displaying student progress over time. To replace the manual 
scoring tool, we developed DeepSign to auto-score students’ videos. The design of DeepSign occurred in two 
phases. Initially, we developed a video data collection tool. We then created a recognition model capable of 
scoring students’ sign language videos and providing immediate feedback. In the following sections, we 
describe PictureNaming and DeepSign in more detail. 


Picture Naming: Student Interface 

PictureNaming measures a student’s ability to sign or say the names of images. We identified 121 
images to use in the measure. Images were selected based on their relevance in sign language or importance they 
have in developing students’ early knowledge. 


Figure 1. Video recording instructions 


The PictureNaming recording process involves using a webcam to capture a video of a user signing the 
meanings of images presented on a computer. Initially, the student locates his/ her torso appropriately in front of 
the camera (see Figure 1). Images are presented to the student continuously for one minute, at progressively 
increasing speeds; initially one image every 5 seconds, then one images every 4 seconds and then one per 3 
seconds (see Figure 2). Progression through the speed levels is determined by the student demonstrating mastery 
on three successive trials. 


Back 


PICTURE NAMING: SHOW ME 


Stert 


Figure 2. Displaying the recorded video 
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PictureNaming: Manual Data Collection Tool and Data Chart 

We designed an interface for teachers to view and score completed PictureNaming tasks (Figure 3). 
After scoring, performance data are added to a database and then to a data chart from which students’ progress 
may be monitored (see Figure 4). 


RABBIT 
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Figure 3. Manual scoring tool 
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Speed 3 (20 images per minute) 


Figure 4. Scoring tasks 


Video Recording Tool 

Our first task was to develop a software tool to capture videos of experts signing the meaning of 10 
images (see Figure 5 for a description of the steps involved in capturing signed input). The tool displays images 
representing core ASL vocabulary (e.g. Apple, Banana, Ice Cream, Milk, Grape, Coat, Cap, Shirt, Pant, Shoe 


316 


(see Figure 7). ASL signers signed the images in front of a webcam. Each image was recorded 5 times to 
provide sufficient variance for a Deep Learning algorithm to recognize similarities and differences within and 
between images (see Figure 6). The 3 signers captured 150 sample videos (see Figure 8). 


PictureNaming Auto-Grading 
Video Data Collection 


First Name: 


Last Name: 


| Practice || Register || Continue | 


*Note 1: For each picture, you have 2 seconds to see the picture and 
another 2 seconds to sign it. 


2: Please Practice first before registering. As soon as you feel 
comfortable. you should press Sign Out and then Register with your 


first and last names. 


3: Please use all of the allocated time to sign. If there is an image of 
an orange, please use all 2 seconds to sign ‘orange’. 


4: Please adjust your camera and lighting to make the video as bright 
as possible. 


5: Please watch the camera while signing. 


6: Please use FireFox (Preferred) or Chome. 


Figure 5. Data collection tool instructions 


Prepare Position 


Progress: 1/605 Begin 
__ User: Practice User 


Figure 6. Data Collection tool 
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Figure 7. PictureNaming Pictures: 
© 2019 SymbolStix LLC All rights reserved. Used with permission. 
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Figure 8. Data Recordings by ASL Signers 


Deep Learning Model 

We used the videos captured by the Video Recording Tool to build a Deep Learning model capable of 
recognizing ASL gestures in real time. The videos were divided into frames that were converted into grayscale 
images. These images were then loaded into a Convolutional Neural Network (CNN) to train the model. A CNN 
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is an artificial neural network commonly used in Deep Learning for image recognition. We chose a CNN over 
other neural networks as a CNN is effective for recognizing visual objects and is memory efficient. It is also less 
complex compared to other Deep Learning neural network models (LeCun & Bengio, 1995; Oyedotun & 
Khashman, 2017). 

The model determines the accuracy of student performance from video input. The model uses 
information about the user’s hand shape, size and color to create grayscale images (see Figure 9). The algorithm 
compares the patterns of input data with patterns in the videos of the data set. The accuracy of the model is 
influenced by the amount of training data supplied to the neural network: the more data supplied, the more 
accurate the model. 


oe 
Figure 9. Setting Hand Histogram for gesture recognition 


Auto-scoring Tool 

We developed a prototype tool (known as DeepSign) to test the recognition reliability of the 
model. The tool presents images and records video while a user signs an object depicted in an image. The Deep 
Learning algorithm converts the video into frames that are compared to previously stored videos. Signs are 
classified as correct when the pixel value of a student’s video is similar to videos stored in the model. 

We used 90 of the 150 videos to train the Deep Learning model and to examine whether the model can 
recognize ASL vocabulary reliably. The other 60 videos were used to test the accuracy of the recognition model. 
The images in Figure 10 illustrate how the model is capable of recognizing differences between images. For 
example, it can recognize small differences between the gestures for the words Apple, Ice Cream and Milk. At 
present, the model can recognize 10 ASL gestures. The rate of successful recognition reached 94.5%. 
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Figure 10. Model Recognizing gesture for Cap, Apple, Ice Cream, Milk 


Discussion 


DeepSign is one of first tool that use Deep Learning to recognize ASL signs. After all training videos were 
analyzed and imported to a CNN, the model was capable of classifying several ASL signs in real-time. The 
results of this pilot study suggest the Deep Learning method has the potential to assess picture-naming tasks 
reliably in a computerized progress monitoring system. The tool will provide students with immediate feedback, 
help teachers monitor student progress efficiently and effectively, and reduce the time for teachers to correct 
students’ work. 

The pilot-study was limited to 10 images from the PictureNaming software. We will extend the 
recognition capabilities of the software to include all 121 PictureNaming images and embed the Deep Learning 
model into the PictureNaming app. Future research will compare the results of auto-scoring with the current 
manual scoring tool and examine whether automating the scoring/assessment process will speed up the rate at 
which students learn ASL. 

We will refine the Deep Learning model to improve scoring accuracy, efficiency, and reliability. Data 
plays an important role in Machine learning. We anticipate the model will continue to improve as additional 
data are added to the system. Also, we will change the image recognition algorithm from the histogram 
approach to the transfer learning- algorithm to reduce the influence of variables such as changing light 
conditions and users’ physical differences (e.g. hand size, shape, and color). 
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Abstract 


This paper reports on the design and development of an intelligent, natural language processing tool, the 
Teacher Responding Tool (TRT), that provides response recommendations to teachers to foster consistent, content- 
specific feedback based on student cognition. Placing student ideas at the center of instructional decisions promotes 
equitable teaching. Results indicate that the TRT selected accurate recommendations and that the interface promoted 
the teachers’ thoughtful consideration of these recommendations. Future design recommendations are provided. 


Introduction 


Research in teacher education shows that placing student ideas at the center of instructional decisions is 
critical for promoting equitable student participation, achievement, and agency (NCTM, 2014). However, 
responding to students in the moment is complex. First, teachers must infer the current understanding of the student 
(Coffey, Hammer, Levin, & Grant, 2011). Second, teachers must prioritize which understandings to focus on for the 
sake of the student, the class, and the intended learning goals (Ball, 1993). Third, the delivery of the response must 
be student-specific, be given in manageable chunks, do more than highlight errors, and avoid comparisons with 
other students (Shute, 2008). Finally, teachers should ask questions that support further student discourse (Chapin, 
O’Conner, & Anderson, 2009). 

Given this complexity, providing teachers with opportunities to develop these skills is important, and there 
have been calls to develop a variety of “approximations of practice” (Grossman, Compton, et al., 2009). This paper 
describes the design and development of an intelligent tool to scaffold teachers’ skills at giving high quality, student 
specific feedback. The Teacher Responding Tool (TRT) is a natural language processing (NLP) tool grounded in 
design principles for worked examples and developing thinking skills (Clark & Mayer, 2016) that provides 
recommendations to support teachers while they respond to students. The TRT builds upon research with 
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technologies that automatically respond directly to students (Aleven & Koedinger, 2002) and that provide teachers 
with insight into student thinking (McDonald, Bird, Zouaq, & Moskal, 2017). 


Theoretical Framework 


This study is informed by instructional design principles aligned with the cognitive theory of multimedia 
learning (Mayer, 2014). These research-based principles describe how to design for learning in contexts that involve 
text and images. While our design is text-based, the assumptions of this theory and many of the design principles 
that follow from it guided our design. For example, this study is aligned with the assumptions that learners have a 
limited capacity to process information, and that learners engage in active processing via selecting, organizing, and 
integrating text. We build upon those design principles that suggest limiting extraneous material, adding cues and 
highlights, using worked examples, and focusing on authentic job-relevant thinking skills (Clark & Mayer, 2016). 


Design of the Teacher Responding Tool (TRT) 


The TRT system consists of three interacting sub-systems: the training dataset creation sub-system, the 
natural language processing (NLP) recommendation engine sub-system, and the user interface sub-system. The TRT 
system is designed to be implemented within authentic learning contexts. To set-up the system for a given learning 
context, two steps need to be taken. First, the training dataset needs to be created and then used to train the NLP 
recommendation engine. The training dataset is created by consulting with the teacher users and collecting their 
prior or suggested responses to prior student explanations for the given context. Second, the TRT needs to be 
connected to the learning management system that the students will be using so that it can pull new student 
explanations after they have been written and push teacher responses back to students. When the TRT is in use, new 
student explanations are pulled from the learning management system and used to select feedback recommendations. 
The recommendations are presented to teachers via the user interface, and the teacher response is then made 
available to the students by pushing these responses to the learning management system. The TRT system is 
designed to include teachers into the feedback process and requires teacher user interaction. As such it is intentional 
that this system does not provide instantaneous feedback to students. Figure | illustrates the subsystems and overall 
system flow. 
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The Teacher Responding Tool System 
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Figure 1. The Teacher Responding Tool (TRT) system. 


Training data subsystem 

Training data creation should aim to use authentic student explanations. In our study we began with student 
explanations collected during a prior implantation of the project within the same context (i.e. same school, same 
course, similar student population). The teacher responses were created by: first, consulting with the teachers about 
how they would respond to each of the student explanations; second, drawing upon research finding for how to best 
provide formative feedback; and third, drawing upon researcher expertise to standardize the responses. This led to a 
training dataset consisting of pairs of student explanation and teacher response written in natural language text. 
Table | shows the first few records of training data used for one of the questions in this study. A different training 
dataset needs to be created for each question prompt. 


Table 1. A sample of the 116 student explanation and teacher response pairs in the training dataset for the question 
prompt: “How does the scale factor change the length, width and height of the object?” 


Student Explanation Teacher Response 


It gets bigger Well done for noticing that there is a change, but what is “it’”? Is the change 
always larger? What happens when the scale factor is 10 or 0.5? 


It makes the size bigger. the Well done for noticing that the dimensions change in response to the change in 


scale factor times by the scale scale factor and that change is "multiplicative", but is this change always 
bigger or increasing? 
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It multiplies the original length Well done for noticing that the length changes in response to the change in the 
by the number of scale factor. scale factor and that the change is “multiplicative”, but what about the other 
dimensions — width and height? 


Length and width and height Well done for noticing that the dimensions changes and that the change is 
multiplied by the 3 "multiplicative", but what change in the scale factor causes these changes? 
What happens to the dimensions when the scale factor is 0.5? 


The bigger the scale factor the Well done for noticing that the dimensions become bigger as the scale factor 

bigger the dimensions get. becomes bigger, but what happens when the scale factor is made smaller rather 
than bigger? 

The new measurements are way Well done for noticing that the size changes, but does the size always increase 

higher than the old in response to change in the scale factor? What happens when the scale factor 

measurements, its shape has = 0.5? What is "it"? Expand on your answer. 

increased. 


We developed training datasets for each question that were on average about 100 records (pairs of student 
explanation and teacher response) long. Generally, larger datasets are known to improve the recommendation engine 
performance. However, factors such as the generality the question being asked and the number of different 
recommendations to choose from also impact how large the training dataset should be (Burrows, Gurevych, & Stein, 
2015; Zehner, Salzer, & Goldhammer, 2016). At the same time, the advantages of larger datasets are offset by the 
time and cost involved in creating them. Based on these considerations, we concluded that about 100 records 
represented a reasonable dataset size. 


NLP recommendations engine subsystem 

The NLP recommendation engine consists of two components: the preprocessing of student explanations, 
and a tf-idf (term frequency-inverse document frequency) model (see Figure 2). The recommendation engine is 
initialized using the training dataset. Each of the student explanations in the training dataset are preprocessed and 
then used to build the tf-idf model. The recommendation engine is used by querying the model with new student 
explanations that have been preprocessed in the same way, and the recommendations selected are outputted. 


Training Data 


Preprocessing 


New Student 
Explanation 


Feedback 
Recommendations 


Figure 2. The natural language processing (NLP) recommendations engine subsystem. 


Preprocessing involves separating a student explanation into individual words (tokenizing) and then 
applying automatic spelling correction to each word. Each word is then converted to its stem, so that words like 
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“increases,” “increasing,” and “increased” all become “increase.” Finally, common high frequency words, or stop 
words, such as “a,” “the,” and “at” are removed. Tokenizing and stemming are performed using the Python n/tk 
package, the spelling correction is performed using the Python autocorrect package, and the stop word list included 
in the Python sklearn package is used. 

The model is built by creating vector representations of each preprocessed student explanation. Weights for 
each word are determined using tf-idf (term frequency-inverse document frequency) which assigns higher weights to 
words that occur less often in the training dataset (Zehner, Salzer, & Goldhammer, 2016). The model is queried by 
finding how similar a new student explanation is when compared to each of the training dataset student 
explanations. Similarity is determined using cosine similarity, i.e. the cosine of the angle between vector 
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representations of student explanations. The teacher feedback in the training dataset that corresponds to the student 
explanations that are most similar to the new student explanation are then examined and the top three unique teacher 
feedback responses are selected as the recommendations. The model building and querying was performed using 
TfidfVectorizer within the Python sklearn package. 


User interface subsystem 

The TRT user interface is an interactive webpage that presents teachers with the question prompt, the 
student explanation, and a teacher-response field (see Figure 3). The three TRT-recommended responses are shown 
in a column on the right side of the screen. When mouse-clicked, the text of the recommendation is copied to the 
teacher-response field, and any text in the teacher-response field can be edited. This allows teachers to use the 
recommendations without making changes, customize a recommendation, take parts of different recommendations, 
or ignore the recommendations and write their own response. 

The layout of the user interface was designed to reduce the extraneous cognitive load that result from 
navigating the page, allowing the teachers to focus their working memory on considering of the recommendations. 
Recommendations were presented near to the student explanations, no scrolling was needed to navigate the page, 
clicking recommendations copy-and-pasted the text, and important text was subtly highlighted. 

The decision to present three recommendations was based on a trade-off between providing teachers with 
enough recommendations to promote the thoughtful consideration of different perspectives, but not too many so as 
to make the cognitive load of the task high and overwhelming. For each recommendation, teachers were expected to 
read the text, consider it in relation to the student explanation, and compare it with other recommendations. From 
this perspective, and considering the length of the text of the recommendations, four recommendations were 
considered by the designer to be the upper limit, two a lower limit, so three were chosen. 


How does the scale factor change the length, width and 
height of the object? 


Student response Recommendations for this 
student 


Click to add to your feecbach Feedback 
New Student Wel dove fer noticing thet the Recommendations 


. length, with and height are 
Explanation changed in response to the change 
in the scale factor, Dut state how 
much is the change? If the scale 
factor is 10 of 0.5, what would be 
the Change in ergt, width end 


Teacher feedback height of the candy? 


Write your feedbeck nere. Or click @ 


Wel Gone tor notcing that te size 
Changes and that the change is 
“mstiolicative’, but does the size 
aways becorne bigger in response 
0 Change in the scale factor? What 


Teacher = a poke a napoens when the scaie factor « 
. —_ os? 
Response 


Wel done tor noticing that the 
Grrersons change - response to 


the change in scale factor, Bis ths 
Change aways Oger or increasing? 
is the charge “multiplicative” or 
“eddiive"? 


Save ard goto rest student 


Figure 3. The user interface subsystem. The teacher user interface displays the question prompt for which the 
training data was collected (top), the new student explanation (top left, in blue), the three recommendations from the 
NLP recommendations engine, and space for the teacher response (bottom left). 
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Method 
Research Questions 


To assess the effectiveness of the tool we asked: 
1. How accurately does the TRT select recommendations? 
2. How effectively do teachers interact with the TRT? 


Data Collection 


Context 

The data for this study was collected in from high school geometry students and their teachers. The 
demographics of the school, located in a rural mid-Atlantic region of the United States, were 12% Black, 44% 
Hispanic, and 38% White students, with 68% of the students receiving free or reduced lunch and 39% of the 
students classified as having Limited English Proficiency. The students participated in a mathematical modeling 
project that focused on how scale factor impacts the dimensions, volume, and surface area of a rectangular prism. 
During the project, the students were asked to write explanations for three different question prompts. The student 
explanations and teacher responses from a prior implementation of the project were used to create the training 
datasets used to answer research question 1. The teacher interactions with the TRT during a subsequent 
implementation of the same project but with different students were used to answer research question 2. 


Research question 1 

In order to assess the effectiveness of the TRT in terms of how accurately it selects recommendations we 
created three training datasets as describe above, with a different dataset for each of three question prompts (see 
Table 2). High school geometry students 


Table 2.4 summary of the three training datasets used to assess the accuracy of the TRT at selecting 


recommendations. 
Training Question Number of student explanation Number of different 
dataset prompt and teacher response pairs teacher responses 
How does the scale factor change 
1 the length, width and height of the 116 29 


object? 

How does the scale factor change 
the volume of the object? 

3 How does the scale factor change 85 6 


the surface area of the object? 


2 99 6 


For each training dataset we performed a leave-one-out cross-validation (Borra & Di Ciaccio, 2010). To do 
this for a given training dataset we first removed one of the student explanations and its corresponding teacher 
feedback. Second, we used the remaining training data to build a tf-idf model as described above. Third, we used the 
removed student explanation to query the model, and finally, recorded whether the recommendations obtained from 
the query included the teacher feedback corresponding to the removed student explanation. If so, we counted this as 
a success; if not, a failure. This process was repeated, leaving out a different student explanation from the training 
dataset each time, until all the student explanations in the dataset had had their turn to be left out. The proportion of 
successes for each training dataset was found by dividing the total number of successes by the size of the dataset, 
and to account for successes that are expected by random chance we calculated kappa for each dataset. A kappa of 0 
indicates that all the success is due to randomness and a kappa of | indicates success every time (Cohen, 1968). 


Research question 2 

To assess the effectiveness of the TRT in terms of how teachers interact with the user interface, four 
classroom teachers were observed using the user interface while following a think-aloud protocol which encouraged 
the teachers to verbalized their thinking as they interacted with the tool. Video screen capture, audio, and researcher 
fieldnotes were recorded throughout. In addition, the four teachers also participated in individual interviews several 
days later with included questions about their use of the TRT. The transcripts of think-aloud and post-project 
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interviews were analyzed to identify reoccurring themes and evidence that confirmed or diverged from the themes. 
The screen capture video was analyzed for the frequency that recommendations were selected or edited by the 
teachers and the time that they spent interacting with the TRT. 


Results 
System Effectiveness 


Research question 1 

After performing leave-one-out cross-validation for each of our training datasets we found kappa values of 
0.51, 0.84 and 0.76 (see Table 3). These results are comparable to those found in other studies that used natural 
language processing technology with open-response items. Liu and colleagues (2014) report average kappa values 
for such studies to be between 0.62 and 0.81. However, the questions considered in these studies only distinguish 
between two and five categories of response, making them more comparable to the question 2 and question 3 dataset 
results rather than the result for the question | dataset which is lower, we expect, due to the high number of unique 
recommendations included in this dataset. Therefore, we conclude that the accuracy of the TRT recommendation 
selection is comparable to those used by other studies. 


Table 3. Number of explanations, unique recommendations, proportion of successful recommendation selections, 
and kappa values by question 


Number of student Number of Proportion of 
Training dataset explanations unique recommendations successful selections Kappa 
Question 1 116 29 0.560 0.509 
Question 2 99 6 0.919 0.838 
Question 3 85 6 0.882 0.764 


Research question 2 

Teachers interacted intuitively with the TRT interface. None of the teachers were observed asking about 
how to use the interface or expressing frustrations with the interface while responding. Instead, teachers were 
positive about their interactions, for example, Henry commented that “it was a very clever interface” and that “it was 
nice to be able to see what [the students] did, try to give a tailored response to give them a hint towards where they 
supposed to be going, and it was also nice to be able to personalize it for them.” 

Teachers interacted thoughtfully with the TRT interface. The teachers interacted with the recommendations 
provided by thoughtfully considering the merits of the different recommendations with respect to the student 
explanation. Mike described that he would “look at the recommendations and think ‘Well, that one clearly isn't what 
I see happening here. This one is the closest to [the student explanation], but I think I need to just qualify it a little 
bit, modify it to fit this situation.”” And Sam said that the recommendations were “something to start off of and 
decide if I agreed with what was there, or if I needed to make up my own.” Nina commented, it was beneficial to her 
that the recommendations were not “everything I want to say as verbatim exactly what I want ... because if it was 
exactly like what I wanted to say, then I feel like [responding to students] would just be a little more mindless for 
me.” 

Thoughtful teacher interactions were supported by the functionality of the TRT interface. The teachers 
thought that the TRT selected the recommendations well. For example, Henry commented that the TRT “generally, 
did a good job pulling recommendations that fit the situation. Many of them I was able to use.” However, because 
the recommendations were often not exactly how a teacher wished to respond to a student, the teachers made use of 
the user interface functionality for selecting and editing the recommendations. As Nina explained, “I could kind of 
pick apart different pieces. It was more of editing, manipulating, or rephrasing what was already given.” The results 
from the analysis of the user interaction data collected from the screen capture video confirm that teachers interacted 
with the recommendations often while responding to students. On average across all teachers, one fourth of the 
teacher responses were unedited recommendations and half of the responses were edited recommendations. At the 
same time, there was some divergence in how the teachers used the recommendations, with Sam mostly writing 
responses without using the recommendations, and Mike mostly using unedited recommendations (see Table 4). 
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Table 4. Average responding time (in seconds) and the use of recommendations in responding, by teacher. 


Number (percentage) of submitted teacher responses that used: 


Average responding no an edited an unedited 
Teacher time (seconds) recommendation recommendation recommendation 
Sam 76.7 5 (56%) 3 (33%) 1(11%) 
Nina 69.6 4 (14%) 21 (75%) 3 (11%) 
Mike 48.2 4 (14%) 7 (25%) 17 (61%) 
Henry 84.7 5 (29%) 10 (59%) 2 (12%) 
Total 66.2 18 (22%) 41 (50%) 23 (28%) 
Discussion 


The Teacher Responding Tool (TRT) was designed and developed to scaffold teachers’ skills at giving high 
quality, student specific feedback. This required that the text of the recommendations aligned with research based- 
practices, that appropriate recommendations were selected for a given student explanation, and that teachers were 
able to interact thoughtfully with the selected recommendations via the user interface. The results from this study 
demonstrate that, for the context in which this pilot version of the TRT was tested, i.e. in the context of high school 
students writing explanations of their understanding during a mathematical modeling project, these requirements 
were satisfied. The TRT was able to select recommendations as accurately as other natural language processing tools 
and, importantly, the teacher users considered the recommendations selected to be appropriate. The user interface 
design supported thoughtful teacher interactions by providing three recommendations, functionality for selecting and 
editing recommendations, and a low extraneous cognitive load layout. A prior study (Bywater, Chiu, Hong, & 
Sankaranarayanan, 2019) demonstrated that thoughtful teacher interactions with the recommendations contributed to 
improved teacher responding practice. This study provides evidence that these interactions were facilitated by the 
TRT design. 


Recommendations 


Several recommendations for designing natural language tools for learning follow from the results of this study. 

1) Natural language processing (NLP) tools that are designed for learning should understand their impact on 
teacher professional skills. In learning contexts, NLP tools are typically used to automatically respond to 
students so that teachers are able to focus their time on those students who are most in need (e.g. Gerard, 
Matuk, McElhaney, & Linn, 2015). In this study, the TRT design included teachers into the responding 
process so that teachers had opportunities to notice how their students were thinking and to develop their 
responding skills. We recommend that designers of NLP tools for learning consider how to they can 
support research-based teacher practices. 

2) Training the system requires specific and purposeful data. The requirement of ‘big data’ is often associated 
with natural language applications and might be thought to limit the applicability of NLP techniques to 
specific learning contexts. This study suggests otherwise. Training a system with a smaller dataset that is 
specific to the applied context can be effective and can support the use in specialized, non-normative, or 
underrepresented learning contexts. 

3) Teacher input into the training dataset content is critical to generate authentic, rich recommendations. The 
process of creating the training dataset might also be considered a novel professional development activity 
that builds upon established practices within the field for teachers collaboratively examining student work. 

4) For tasks that are cognitively demanding, extraneous cognitive load can be reduced by using a minimalist 
user interface design that retains all necessary functionality. In this study, reducing extraneous cognitive 
load involved both how to best present information and how to reduce the load associated with interacting 
with the information. The TRT user interface combined the select-copy-select-paste steps that a user 
frequently repeats into a single click. This simplified the interaction steps for teachers when selecting and 
editing. 

5) Connecting the TRT with different learning management systems requires permissions to share identifiable 
user data and technical expertise. These challenges continue to present adoption hurdles but are being 
addressed within the educational technology community (e.g. Learning Tools Interoperability, 2019) and 
we recommend common standards and protocols to mitigate these challenges. 
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Abstract 


This paper is intended to assist full and part-time faculty and administration on training and inclusion into 
the educational institution. Part-time instructors, sometimes referred to as adjunct instructors, are a vital part of 
higher education institutions. In the United States, approximately 62% of the faculty at degree-granting institutions 
are employed on a part-time basis (Ruiz, 2015). Proper training on school policies, online management systems and 
pedagogy for part-time faculty is imperative due to the increasing number of these faculty members in higher 
education institutions. Part-time instructors should be connected to the institution and the department that they are 
teaching. The better connected one feels, the more likely they are to have job satisfaction and will retain their 
position, which provides students with continuity in their education. 


Introduction 


Part-time instructors, sometimes referred to as adjunct instructors, are a vital part of higher education 
institutions. In 1969, approximately 21% of college faculty in the United States were employed on a part-time basis 
(Tolley, 2018). As of 2015, approximately 62% of the faculty at degree-granting institutions are employed on a part- 
time basis (Ruiz, 2015). With the ever-present economic considerations in the field of education, it is likely that this 
number will increase. When the economy declines, higher education administrators reduce costs by hiring adjunct 
faculty (Tolley, 2018). That increase is just one of the reasons it is vital for adjunct faculty to be properly trained on 
school policies, online management systems and pedagogy within their respective departments and feel as though 
they are active members of the academic community. 

Proper training in pedagogy, content and technology is also imperative because with the increasing number 
of adjunct faculty, means that more college students are being taught by part-time instructors. With the lack of 
funding for higher education, there is a deficiency in the amount of training that part-time faculty receive. Some 
colleges and universities have a general training program for the online management system that they use, but aside 
from that, adjunct faculty are often left with little or no support. This is problematic for several reasons, but 
primarily because some of these adjuncts may not have any previous teaching experience. 

While most educators might agree that adjunct instructors have a wealth of content knowledge to share, it is 
a disservice to the students if the instructors are not properly trained in how to best disseminate that knowledge. 
Adjunct instructors also need to be connected to the institution and the department that they are teaching. They 
should feel encouraged and supported. The more connected to an institution, especially to a specific department, an 
adjunct instructor feels, the more likely they are to be successful in the classroom, both online and face to face. This 
responsibility lies with the full-time faculty members and administration of the institution at which they are 
teaching. “They should be included in all college activities that deal with effective teaching and should have 
opportunities to connect with colleagues” (McGlynn, 2014, p.19). 


Suggested Strategies 


There are a variety of ways to incorporate part-time faculty into the day-to-day activities of the institution. 
Even if part-time instructors cannot participate due to other employment obligations, they will likely still feel valued 
because they were included. One simple way for inclusion of adjunct faculty is to invite them to department faculty 
meetings and any school-wide employee appreciation or in-service days. This approach is also cost-effective, since 
the part-time faculty would be an addition to an already scheduled event. This is especially important for online 
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faculty who may not be required to visit the campus for classes. Even if part-time instructors are unable to attend, 
they are likely to feel valued just for the simple gesture. 

Some colleges have begun paying their part-time faculty to take professional development courses (Anft, 
2018). There are others that have on-campus staff development days where part-time faculty are encouraged to 
attend. Even if the institution does not have the funding to pay for part-time faculty to take courses or attend 
conferences, it would be easy to send an email notifying them of upcoming events. 

Another example of including faculty where they would also be gaining knowledge on how to teach, is for 
part-time faculty to observe full-time faculty teaching their courses. The part-time instructors could then share with 
their supervisor what they learned, and what components will implemented into their course(s) (Anft, 2018). This 
could be a favorable approach for new part-time instructors who do not have previous teaching experience. 

Many institutions have student clubs in which adjunct faculty can get involved. For example, the largest 
technical college in the state of Georgia, Chattahoochee Technical College, has a Public Safety Club. This club is 
run by the Lead Instructor of the Criminal Justice department as well as another full-time instructor. The club 
consists of student members who meet bi-weekly, have guest speakers, and take day trips to police departments, jails 
and prisons. This gives the students a different perspective into the criminal justice system as well as an excellent 
opportunity for networking. Inviting an adjunct instructor to be a guest speaker, either at a club meeting or ina 
regular class meeting, is a great way to make them feel a part of the college and also an opportunity for them to meet 
some of their students or potential students in person. After all, part-time instructors are the subject matter experts 
who are currently working in the field that they are teaching. This training is especially important for adjunct 
instructors who teach solely or primarily online. Face-to-face interaction is a vital component to a positive 
relationship. With over half of college instructors being part-time, training is imperative, as is providing 
opportunities to participate in college activities. Ensuring that instructors feel inspired and supported, will provide a 
positive learning environment for students. 


On-boarding Process 


The author has been an educator for 10 years and began as an adjunct instructor. Over the years, after 
working with several part-time faculty, training guidelines have been created to make sure that the most important 
topics are covered. When a new adjunct instructor is hired, the author proposes the following steps be taken to 
ensure that they are adequately trained and comfortable with their new role in the classroom. After all, part-time 
instructors are the professionals working in the field, but may need some guidance in how to effectively distribute 
the information to students. 

1. Pair the part-time instructor with a faculty member in the department where they will be teaching. 

2. Provide them with all relevant materials (syllabus, previous assignments/tests, learning outcomes for 

the course, etc.) and explain the importance of timely feedback and frequent checks for understanding. 

3. Meet with the instructor to train them on the online learning system. This meeting should be at their 
convenience and might have to be during non-traditional hours and/or at their place of employment. 

4. Invite them to observe a face-to-face class so that they can get some familiarity with what will be 
expected of them. 

5. Get the instructor in contact with the book publisher for the course that they will be teaching. 
Providing this contact information will allow the instructor to communicate directly with the 
publisher’s representative so that the instructor can receive all online supplemental resources. 

6. Give the instructor a hard copy as well as an electronic copy of “important” people that they might 
need to reach at the college (Human Resources, Information Technology etc.). 

7. Invite them to an upcoming meeting or event that the school is hosting. This should be done 
throughout their employment and not just when they are first hired on. 

8. Ensure that they are familiar with the policies and procedures of the school. This includes training on 
ADA accommodations, sexual harassment policies and plagiarism. 

9. In addition to the hands-on training, provide them with a hard copy of administrative duties and how to 
perform them. For example, how to submit final grades. 

10. Demonstrate how to create videos to embed into their courses and show them online resources that will 
aid in teaching the material (Kahoot, Quizlet, etc.). 


This training meeting is an important time to get to know your new instructor and develop a relationship 


and an open line of communication. Part-time instructors “may not be cognizant of the constantly evolving variety 
of teaching methods or proficient at implementing them” (Strom-Gottfried & Dunlap, 2004, p. 446). New instructors 
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are likely to feel overwhelmed with all of the new information that they are learning, and a little apprehensive if they 
are new to teaching. To reduce any possible nervousness, follow up with an email outlining everything that was 
covered during that meeting. It is good to have a resource to refer back to at any time. 


Conclusion 


Students deserve a high-quality education at any institution. With more than half of higher education 
faculty being part-time, their role is critical to the college or university employing them. Faculty development and 
training should be part of every new instructors’ goals and it is the responsibility of the individual institution to 
provide that training to them. If part-time instructors feel dissatisfied with the educational institution due to lack of 
training and support, and as a result, do not maintain their employment, this is a disservice to our students. 

Full-time faculty should embrace part-time faculty because they are an asset to our students learning 
experience and aid in reducing large class sizes and teaching overloads. 
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Abstract 


The purpose of this paper is to review the implementation and utilization of makerspaces to inspire the 
development of problem-solving skills within K-12 learning. There is consensus regarding the need for K-12 
education to meet the needs of the 21st Century learner. Foremost among these needs, according to the Partnership 
for 21st Century Schools (2008a) are critical thinking and problem-solving skills. Based on the review of research, 
the authors present a brief history of the ways in which problem-solving and critical thinking have been addressed in 
the K-12 classroom in the past, and conclude with a description of how integrating makerspaces through STEM 
learning can impact and help develop these learning skills in order to meet the needs in the future job market. 


Inspired to Make 


There is, in the world of education, much discussion regarding the implications of the 21st Century society 
and the needs of the 21st Century learner. The evolution from the Industrial Age to the Information Age requires our 
education system to meet the needs of these 21st Century learners as they prepare for a society and economy very 
different from that of the past. Foremost among these needs, according to the Partnership for 21st Century Schools 
(2008a) are critical thinking and problem-solving skills. Researching the history of the ways in which problem- 
solving and critical thinking have been addressed in the K-12 classroom in the past provides us a glimpse of a path 
which leads from orderly lines of desks, restraining students as they memorize and regurgitate information — into a 
learner-centered, flexible workspace of today, where creativity and innovation work hand in hand developing 
technical knowledge and skills. Integrating makerspaces through STEM learning activities can help develop critical 
thinking and problem-solving skills in order to meet the needs of the next generation. 


Critical Need for Creative Thinking and Problem-Solving Skills 


Critical thinking, creativity, collaboration, and communication, also known as the 4C’s, are regarded as 
some of the most important skills of the 21st century (Partnership for 21st Century Skills, 2008a). The significance 
of critical thinking is paramount in education, and its connection to the other 3Cs is equally valued. Leading experts 
on critical thinking stress the importance of its connection to creativity as well (e.g., Paul & Elder, 2006). Critical 
thinking can be broadly defined as “a developmental process that proceeds from experience (e.g., observation and 
interaction) to inquiry, investigation, examination of evidence, exploration of alternatives, argumentation, testing 
conclusions, rethinking assumptions, and to reflecting on the entire process” (Ma et al., in press). 

Spector and Ma (2019) presented a framework of critical thinking with four dimensions: dispositions, 
skills/abilities, levels and time. Besides those dimensions, there are other essential elements of critical thinking such 
as motivation, criteria, domain knowledge, context, etc. The framework interpreted critical thinking from a holistic 
viewpoint and emphasized the dynamic nature of critical thinking in terms of a specific context and a developmental 
approach. The National Education Association (2011) defined critical thinking in four phases: to reason effectively, 
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use system thinking, make judgements and decisions, and solve problems. Solving problems successfully and 
effectively requires a process of thinking critically. Simply put, the relationship between critical thinking and 
problem solving can be stated as follows, “Students who are able to think critically are able to solve problems 
effectively” (Snyder & Snyder, 2008, p.90). Developing problem-solving skills has been identified as essential to 
student success in the 21st century, not only inside the classroom but also in all professions (Jonassen, 2010; 
Kereluik, Mishra, Fahnoe & Terry, 2013). 

Problem-solving skills have been researched for decades and the term problem-solving has been defined in 
various ways. For example, Kendler and Kendler (1962) believed that instead of being conceptualized as a basic 
psychological process, problem solving can best be conceptualized as one that reflects the interaction of more 
fundamental processes (e.g., learning, perception, and motivation). D'zurilla and Goldfried (1971), defined problem- 
solving as a behavioral process that provides different responses to problematic situations and increase the 
probability of success in effectively solving problems. They identified five stages of problem-solving: 1) general 
orientation, 2) problem definition and formulation, 3) generation of alternatives, 4) decision making, and 5) 
verification. 

Problem-solving requires the ability to draw from previous experiences so the learner can apply what they 
already know from various content areas. Therefore, training students in problem-solving will mean educators need 
to allow them to "learn how to solve problems" and thus discover the most effective way to individually respond. An 
investigation by Phonapichat, Wongwanich, and Sujiva (2014) revealed that there were difficulties with elementary 
students’ mathematical problem-solving which included: 1) difficulties with understanding some keywords, thus 
they were unable to interpret them in mathematical sentences, 2) students were unable to figure out what to assume 
and what information from the problem was necessary to solving it, 3) students tended to guess the answer without 
going through any thinking process, 4) students were impatient and did not like to read mathematical problems. 
Those difficulties indicated the lack of robust problem-solving skills or the necessary training of critical thinking 
among those elementary students. Ultimately, the study verified that certain domain knowledge is necessary for the 
problem-solving process. 

Just as elementary students struggle to learn problem-solving skills, it is evident that the lack of growth 
transfers through middle and high school as well. According to a report from Partnership for 21st Century Skills 
(2008a), 76% of the business leaders of U.S. companies surveyed indicated the significance of problem-solving 
skills and creativity and their inevitability to increase considerably in the workforce as the problems in need of 
solutions grow evermore complex. Additionally, those same leaders reported that there was a lack of critical 
thinking and problem-solving skills among hired high school students (Partnership for 21st Century Skills, 2008b). 

In the recent standards of science education, critical thinking and problem-solving skills are emphasized 
(National Research Council, 2013). Through the process of problem solving, learners engage in critical thinking 
with the aid of creativity and design. Scientific investigation and engineering design are placed at the forefront of 
science and engineering learning in the next generation science standards (National Academies of Sciences, 
Engineering, and Medicine, 2019). Just as Silva (2008) notes “it is an emphasis on what students can do with 
knowledge, rather than what units of knowledge they have, that best describes the essence of 21st century skills” (p. 
2). In order to prepare for the critical thinking jobs of the future, it is important to understand where the education 
system derives and how to continue molding and improving to fill the needs of the future job market. Unfortunately, 
there are very few states in the United States which have adopted or partly adopted the next generation science 
standards. 


History of Problem-Solving in the Classroom 


Within the U.S. education system, the skill of problem-solving has been addressed in different ways and 
different teaching and learning methodologies have been utilized to promote success. In order to appreciate the 
development of problem-solving skills in the classroom, it is important to review the progression of classroom 
structure and methods for teaching over the past half century. Starting with the “traditional classroom”, one can 
identify additional tools and methods that have been designed and added to advance critical thinking in the 
classroom setting over the years. These tools and methods include problem-based learning, STEM, STEAM and 
makerspaces. 


The Traditional Classroom 

To understand today’s classroom setting in U.S. schools, one must dissect how current learning is achieved 
and what might be missing in order to improve future educational practices. When researchers refer to the U.S. 
education system, there is often mention of the “traditional classroom,” yet that terminology is not always defined. 
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One descriptive definition found in literature (Hertz-Lazarowitz, 1992) describes the traditional classroom as a 
“single social system” (p. 73), with the teacher presenting herself as the center of activity, while students are isolated 
in rows of individual desks. The teacher “controls all communication networks and presents knowledge to 
pupils...The learning task is structured as individualistic or competitive...Student-to-student interactions are minimal, 
and each student looks after himself or herself.” (Hertz-Lazarowitz, 1992, p. 73). While this type of learning was 
important during the industrial revolution in order to reach and teach the masses, the educational practices have 
changed little since then. This is despite that fact that digital technologies are changing not just how we live and 
work, but how we think and behave (Keegan, 2012). Additionally, researchers continue to impress upon educational 
leaders that teaching must advance and change from a teacher-centered to a student-centered experience (Froyd, 
2008). 

As a teaching method common throughout the 20" century, rote memorization appears in discussions of the 
traditional classroom, and, although there are arguments for the purposeful application of memorization in education 
(Caron, 2007), there are many voices urging the importance of escaping the perceived outdated reliance on rote 
memorization as the very definition of education (Canestrari, 2008; Fata-Hartley, 2011; Snyder & Snyder, 2008). 
Rote-memorization dates to Confucian education from early China and Tan (2015) refers to it as a form of surface 
learning as opposed to a deeper learning method. Surface learning or reproducing content is essentially the opposite 
of critical learning and thinking, which requires what is referred to as deep learning, or an “intention to understand 
content in order to relate and structure ideas” (Dolmans, Loyens, Marcq, & Gijbels, 2016). As the evolution of 
transformation in the classroom continues, so should the evolution of learning. Education should emerge from a 
method of surface learning and rote-memorization to a deeper learning structure integrating the critical thinking and 
learning processes laid out in 21 century skills framework. 


Problem Based Learning 

Problem based learning (PBL) is an educational method that introduces relevant real-world problems in the 
classroom in order to encourage deeper learning and critical thinking to design applicable solutions (Dolmans et al, 
2016). In 1969, McMaster University first developed curriculum using problem-based learning as part of its medical 
school (Banta, Black, & Kline, 2001). In recent years, the Harmony Public Schools developed a curriculum that 
incorporates project-based learning into STEM called “STEM Students on the Stage (STEM SOS)” (Sahin & Top, 
2015). The Harmony Schools curriculum was designed to help teachers by providing structure to what is often 
perceived as an unstructured teaching method. 


STEM 

For the last two decades the term STEM has become the focus of educational practitioners and researchers 
all over the globe. STEM was coined in 2001 by leaders at the National Science Foundation to describe curriculum 
that involved four different areas of learning: science, technology, engineering, and math (Patton, 2013). In 2011, 
the concept became more important with nationwide implementation through the federally funded initiative called 
Educate to Innovate. According to Ostler (2012) the problem that existed with this initiative was that all parties 
involved “markedly under-conceptualized what STEM education was” (p. 28) and how schools could and should 
implement the initiative. As a result, researchers have spent the past decade studying and learning effective ways to 
implement STEM learning. The U.S. Department of Education’s (USDOE, 2018) website states, “If we want a 
nation where our future leaders, neighbors, and workers have the ability to understand and solve some of the 
complex challenges of today and tomorrow, and to meet the demands of the dynamic and evolving workforce, 
building students’ skills, content knowledge, and fluency in STEM fields is essential.” In fact, in 2018, the U.S. 
Department of Education surpassed President Trump’s request for directing $200M to STEM learning, by investing 
$279M in discretionary grant funding (USDOE, 2018). 


STEAM 

Although the acronym STEM is wide-spread and is argued by many as encompassing all areas of learning, 
by 2018 there was a movement in education circles which argued for the addition of the letter A for art to create the 
word STEAM (Chu, Martin & Park, 2018). It is argued that the addition of Art into traditional STEM addresses the 
importance of creativity in education (Conradty & Bogner, 2018). Beyond that, there are those who argue that, when 
Art is integrated into STEM, “new understandings and artifacts emerge that transcend either discipline” (Peppler & 
Wohlwend, 2018, p. 1), thus enriching multiple content areas. For the purpose of this article, we will acknowledge 
Gerlach’s (2012) concern with the fact that STEM “means so much for so many different groups of people” (para 9) 
- and use the acronyms STEM and STEAM interchangeably to focus on the common overall concept of using an 
interdisciplinary approach to solve problems within the realm of science, technology, engineering, and math. 
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Maker Movement and Makerspaces 

There are those who claim that today’s maker movement started with the inception of Make Magazine 
(Halverson & Sheridan 2014), while others would argue that the foundation of the maker movement can be dated 
back to Seymore Papert’s constructionism, the theory that highlights the process of learning to learn and making 
things to help solidify learning (Pernecky, 2012). Regardless of when it started, the last two decades have shown a 
surge of research in the topic area that was virtually non-existent prior to the 21“ century. The idea of Making or 
makerspaces over the last two decades have been referred to and researched under several different names including 
hackerspaces, fab-labs, and tinkering. Although other names exist, they all refer to the act of investigating, creating, 
remixing, designing, redesigning, fixing or tinkering in order to create and/or understand something (Moorefield- 
Lang, 2015). This paper will refer to the creative act of Making by capitalizing the term to help identify the process 
for which it is named. The result can be the creation of something that already exists, or it can be something entirely 
brand new. While collaboration may play a part, it doesn’t have to be present in the process. The maker movement 
allows for a free flow of ideas from anyone who wants to take part in the Making, even if it is only just one person. 
As the maker movement gained popularity, President Obama presented the first White House Maker Faire in 2009 
(Halverson & Sheridan, 2014). Obama focused interest on the educational community by saying “I want us all to 
think about new and creative ways to engage young people in science and engineering, whether its science fairs, 
robotics competitions, fairs that encourage young people to create and build and invent, to be makers of things, not 
just consumers of things” (Obama, 2010, p. 561). This type of vision can encourage a whole new generation of 
tinkerers and makers to develop and flourish and reinforces reason to study the impact of this creative learning 
methodology. 


Maker Movement’s Role in Critical Thinking and Problem-Solving Skills 

Eleanor Roosevelt, in her book Tomorrow is Now (1963), shared: “It is one thing to provide a simple skill 
that can be applied to a given situation. It is quite another thing — and a new, a revolutionary thing — to prepare 
young people to meet an unknown world...” (p. 66). Understanding the history of problem-solving and critical 
thinking in education allows researchers to better design a course of action for improving learning for the future. 
Knowing what has worked historically — and where the shortfalls are — creates a path for others to navigate and 
design new methods for learning and developing future generations of critical thinkers. Inspiring students to 
problem-solve in a makerspace using the problem-based learning approach to solve real world STEM problems is 
just one of the ways makerspaces have invigorated the role of the learner and helped educators and scholars to see 
unique ways in which to challenge students to become natural problem solvers (Slatter & Howard, 2013). 


Makerspaces by Design 

Makerspaces can be designed in a multitude of ways. One of the best parts of makerspaces is that it is 
created to benefit the community in which it is placed, so each one can look uniquely different (Oliver, 2016). 
Community and school libraries host the greatest number of makerspaces, as their design by nature lends itself to the 
activity (Colgrove, 2013). There are many reasons why libraries and makerspaces go together. These include: 1) 
creating purpose for visitors to come in and use the facilities, 2) their wealth of resources enhances the learning 
before, during and after any build, 3) community experts have a place to share their knowledge with others, and 4) 
they enhance community outreach (Slatter & Howard, 2013). While communities may have a host of other, 
additional, reasons to implement a makerspace, each reason provides an opportunity to help develop the people and 
the community. 

Schools have also joined in on Making and started implementing makerspaces to increase student 
engagement, enhance problem-solving skills, and to improve collaborative learning opportunities. When 
implemented in schools, the designers of the spaces have created all types of unique spaces for their students. In 
some elementary spaces you may see Lego walls or tables (Medlar, 2016), computers of various sizes and shapes, or 
even brown paper bags full of elements such as straws or toothpicks, cotton balls or bottle tops — often with a 
challenge written right on the front of the bag. These tools may be mobile in order to conserve space and allow for 
ease of display when needed for use. 

In middle schools and high schools, you are likely to see larger spaces and often permanent ones that are 
set up in a classroom or, more often, in the library. The spaces usually have flexible seating for easy movement and 
collaborative working. Tables are typically taller to accommodate both standing or sitting on a tall stool. Often, there 
are computers and 3D printers (Moorefield-Lang, 2015) flanking the space and research books to aid in learning. 
These spaces usually have simple tools, such as scissors, markers, poster board or poster paper and more complex 
tools like switches, motors, circuits and even robotics of various types and brands (Oliver, 2016). Librarians will 
keep old or retired books in the space for recreating new works of art. There are often Arduinos (Gustafson, 2013) 


337 


for the building of electronic pieces that can do or accomplish things with computer programming. If messiness is 
allowed in the space, like a mess one might find in a shop, the space may hold welding tools or woodworking tools 
for bigger builds. Each space is designed specifically for the students it serves with the budget available or set aside 
for that school or space. 

Multiple makerspaces to accommodate different situations is even likely in some locations. Many of the 
larger colleges don’t stop at just one makerspace. At many universities, you may find upwards of five or six spaces 
around campus. Colleges like the University of California at Berkeley, University of Washington and Princeton all 
boast about the size or multitude of makerspaces around their campuses. Colleges are even seeing the value in places 
and spaces for innovative design and collaborative creations. Some of the greatest creations came from garage 
makerspaces before their time. Consider the beginning of Microsoft and Apple. Makerspaces can produce inventions 
and discoveries whether it is in a school or somewhere else in a community. These spaces drive innovation and 
allow the mind of any age to think deeply and critically while developing a solution to what likely started as a 
simple question: I wonder...? 


Problem-Solving in Makerspaces 

Cooper and Heaverlo (2013) found that problems and their potential solutions are a common driving force 
in motivating students to want to learn. While motivation is important, they also discovered that collaboration is 
another key to problem solving, which lends itself to the design of makerspaces. With the collaborative design of a 
makerspace it becomes a natural environment to create cross-disciplinary problem-solving activities which can 
“foster higher levels of epistemic and intellectual development in students” (Cooper & Heaverlo, 2013, p. 27). 

In many educational circles today, Making is employed as a major approach to developing problem-solving 
abilities among children. However, regardless of approaches adopted to develop problem-solving skills, it is not a 
simple idea. It involves conceptual knowledge and procedural knowledge, and consequently there are rules for the 
development of these skills, according to Riley (1984). With age, children's improved ability to solve word problems 
primarily involve an increase in the complexity of conceptual knowledge required to understand the problems’ 
context. Therefore, the development of conceptual knowledge requires the consideration of the sequences of 
different types of problem-solving skills and one’s progress should be aligned with the stages of child development. 

In order to encourage children’s problem-solving abilities, there are many developed and well-established 
approaches that have been researched. However, current scholars in K-12 learning specifically call on the movement 
of Making. Martin (2015) argued that “Making gives youth access to sophisticated tools for building and for 
thinking” (p. 36). One example is the increasing awareness and development of creativity among children - 
especially in STEM fields. Taylor (2016) evaluated the benefits of the maker movement in K-12 education and 
believed that Making allows students to connect the hands and the mind in a cognitive and physical capacity, 
preparing them with 21st century skills. With the understanding of the research and a clear pathway for learner’s 
success, Making can be the cornerstone of future educational approaches to critical thinking and problem-solving. 
Sheffield, Koul, Blackley, and Maynard (2017) argued that “a makerspace approach to STEM education can be an 
authentic and robust pedagogical practice providing there are strong and explicit connections the curricula of 
mathematics, science and technology and the resultant [makerspace] product...” (p. 152). As is the case with any 
new tool, only if the makerspace is implemented effectively will it enhance learning and help to develop the desired 
critical thinking and problem-solving skills. 


Implementation of STEM Makerspaces 

One of the most time consuming and difficult parts of implementing any new learning practice or tool for 
teachers is creating effective strategies for teaching and learning Bevan, Gutwill, Petrich, & Wilkinson, 2014). With 
makerspaces, it becomes even more difficult because teachers often still have the traditional classroom mindset and 
must guide and develop the student’s abilities to a point that they can be released to create and discover on their own 
(Bevan et al., 2014). It is the release of ownership from teacher to student that allows the student to develop the 
critical thinking skills necessary to meet those standards needed in the future workplace. As students begin to 
develop the confidence in their own confusion and struggle, to, ultimately, discovery and success, their confidence 
helps them to pursue additional goals with less hesitation. When this process can be achieved in an engaging activity 
such as those a makerspace can offer, students become eager to learn and to keep moving forward. When teachers 
can see success in students, they, too, can become more invested in the design of makerspaces and their abilities to 
guide learning without being the center of learning activities. As the students learn through gradual ownership, 
teachers learn through the process of gradual release of responsibilities. 

Bevan et al. set out to design a framework to help teachers and faculty effectively implement tinkering 
activities into the classroom curriculum (2014). The team consisted of researchers and practitioners so that research 
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supported techniques and practical application could join to create an effective plan for the future of makerspaces in 
communities and schools. The resulting framework is widely used today by practitioners in the educational field and 
beyond. The study focused on tinkering in the area of STEM learning through afterschool programs. What resulted 
was four dimensions of learning: “engagement, initiative and intentionality, social scaffolding and developing 
understanding,” (Bevan et al., 2014, p. 106) The team concludes that there is great potential for tinkering as a tool 
for learning (Bevan et al., 2014). While they know that theirs is just one tool for capturing learning, it is a solid 
researcher and practitioner-based start for STEM and makerspaces and an engaging opportunity for future critical 
thinkers. 


Conclusion 


We live in a time where people often discover success purely by accident. To believe that we can predict 
what tomorrow will bring in the way of successful careers and opportunities would be misguided. As researchers 
and practitioners in the educational field know, success often depends on the simple act of trying and then trying 
again. That saying often holds true in the implementation of, and activities within, makerspaces as well. If we do not 
attempt to find new and innovative ways to inspire future generations moving through the educational system, the 
outside world will continue to be more interesting and the world within the school walls will become irrelevant — 
and simply put, a means to an end. In order to inspire discovery and meet the dire need for critical thinking, it is 
important that we help learners develop the problem-solving skills needed so that when faced with new 
opportunities for growth in learning, they become inspired to move forward with the challenge. 

As STEM learning allows for a cross-curricular training in unique skillsets, so too does the marrying of 
STEM and makerspaces for learning those skills. These environments are safe places for students to attempt 
creation, to fail, and to collaborate with others in the reattempt for success. Without opportunities like these, we 
often find the learner and the educator stuck in the rut of traditional classroom learning. It is the authors’ intent to 
bring attention to the marrying of STEM and makerspaces to engage budding problem-solvers and help them to 
develop the needed skills for tomorrow’s careers. It is also our hope to encourage teachers to challenge their own 
abilities with the implementation and integration of cross-curricular spaces for student Making. Together, we can 
build opportunities and experiences that will encourage the next generation of students to go beyond their comfort 
zones, be excited to learn, and more specifically, inspired to Make. 
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What Is Interest? 

What is interest? How is interest utilized, nurtured, and promoted in the STEM (Science, Technology, 
Engineering, Mathematics) social constructivist classroom? How is interest related to STEM learning outcomes? 
(Krapp, 2005). New interests are possible at any time during any stage of a person's life (Renninger & Hidi, 2016), 
but the flipside of that is students can also lose interests at any time. This paper will explore the nature of interest, 
expertise, and identity in learning, especially in a social constructivist STEM classroom or school library. 
Pedagogical recommendations are offered to educators and instructional designers based on viewing learning 
through the lens of interest development in a social constructivist environment. 

Interest drives our vocations and avocations. It influences what we do and enjoy at home, at school, at 
work, at play, at love and at war. Renninger & Hidi (2016) define interest as the state in which, when people interact 
with an activity, they “. .. voluntarily engage in thinking about it, happily prioritize the problems that arise . . . and 
are willing to persevere to address them” (p. 1). People are hardwired for interest, and that tendency does not change 
throughout their lives (Renninger & Hidi, 2016). Interest has two components. The first refers to a person’s 
psychological state while he or she is engaging with content, while the second is connected to a more longstanding 
affective and cognitive desire to reengage with the same content over time (Krapp, Hidi, & Renninger, 1992; 
Renninger & Hidi, 2016). The first type of interest originates from outside of the individual and is called extrinsic or 
situational interest, while the second type, intrinsic or individual interest, originates within the individual. Person X 
has recently become engaged with zines because he thinks they are cool (situational interest, triggered by context- 
rich zines and/or sharing and encouragement from friends), while Person Y is an avid collector who spends much of 
her free time reading and trading zines (individual interest, triggered by her frequent previous engagement and long- 
standing, context-rich relationship with zines and their communities)—both people are interested in zines, but 
Person Y expresses a deeper, richer, and longer-lasting level of interest than Person X. Those levels are further 
defined in the Four Phases of Interest Development (Hidi & Renninger, 2006) (see Table 1 below). 

Empirical research on interest triggers has mainly consisted of experiments using specific novelties or 
challenges, referred to here as triggers (Renninger & Bachrach, 2015). In the early stages of interest, perhaps for 
Person X, learners may not even know that their interest has been triggered; the power of interest is manifested 
when behavior changes based on the interest (Renninger & Bachrach, 2015). However, triggered situational interest 
can also be fleeting (Renninger & Riley, 2013; Renninger & Hidi, 2016); a volcano eruption, author visit, or worm 
dissection can be exciting today but forgotten tomorrow (Crouch, et al, 2018). Additionally, seductive details, 
distracting information and/or events that deter learning (Garner, et al, 1992; Wade, 1992; Wang & Adesope, 2016), 
are ready to steal learners’ attention with misreadings, ancillary facts, too much focus on graphics in a text or 
documentary, or even changeable weather. 

In deeper interest levels, learners provide their own triggers (Renninger & Bachrach, 2015; Renninger & 
Hidi, 2016); they persevere with the task and even see themselves performing the task as part of their career 
(Renninger & Riley, 2013). However, at lower levels of situational interest, engagement and involvement may be 
fleeting without personal connection and perceived value, often manifested merely as “interestingness” (Hidi & 
Anderson, 1992; Patall, 2013). Although people tend to have 4 or 5 well-developed interests, there is always room 
for shifting and changing; focus on interest changes regularly (Renninger & Bachrach, 2015; Renninger & Hidi, 
2016; Renninger & Riley, 2012). Engagement is an element of interest, but they are not interchangeable. One can be 
engaged but not interested, but one cannot be interested without being engaged (Renninger & Bachrach, 2015). 
Engagement in this respect can be seen as some level of involvement in an activity, a prerequisite for interest, but 
that is in this educational context only; the field of engagement research is not addressed here. 
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Interest can represent both a psychological state experienced in a particular moment situationally, and an 
enduring predisposition to reengage with an object or concept/topic over time (Harackiewicz, Smith, & Priniski, 
2016). The state of being interested is experienced as flow (Csikszentmihalyi, 1990; Renninger & Hidi, 2016) and if 
maintained, interest leads to enhanced academic performance, or maybe to enhanced performance across many 
measures. Interest must be consistently nurtured to be retained, and students in different stages of interest 
development along the Four-Phase Interest Development model (see below) require different types of interventions, 
featuring personalization and authenticity whenever possible. Harackiewicz, Smith, & Priniski (2016) offer three 
dimensions of personalization: 

Depth—quality of the personal connections to learners’ interests; how authentic to the student are uses of popular 
and personal topics; 

Grain size—size of the reference group; is it individualized instruction for an IEP (one student or a couple of 
students), or is it the type of individualized instruction a teacher would offer in a certain situation (many students); 
Ownership—the degree of autonomy (Ryan & Deci, 2017) in generating the personalization. As expected, the more 
students self-determine, the deeper their connection gets to the content, and, it is hoped, the higher they rise in the 
Four Phases of Interest Development. 

Interest development is a rich field that has shifted considerably since the publication of the Four Phases of Interest 
Development (Hidi & Renninger, 2006). However, the possibility to be interested in STEM is not available to all 
students. 


Table 1. Hidi & Renninger’s (2006) Four Phases of Interest Development. 


Less-Developed (Earlier) More-Developed (Later) 
Phase I: Phase 2: Phase 3: Phase 4: 
Triggered Situational Maintained Situational __ Emerging Individual Well-Developed 
Interest Interest Interest Individual Interest 


e Psychological state e Psychological state e Psychological state e Psychological state 


Definition 


Learner Characteristics 


resulting from short- 
term changes in 
cognitive and affective 
processing associated 
with a particular class 
of content 


Attends to content, if 
only fleetingly 

May or may not be 
reflectively aware of 
the experience 

May need support to 
engage from others 
and through 
instructional design 
May experience either 
positive or negative 
feelings 


that involves focused 
attention to a 
particular class of 
content that reoccurs 
and/or persists over 
time 


Reengages content 
that previously 
triggered attention 

Is developing 
knowledge of the 
content 

Is developing a sense 
of the content’s value 
Is likely to be 
supported by others 
to find connections to 
content based on 
existing skills, 
knowledge, and/or 
prior experience 

Is likely to have 
positive feelings 


and the beginning of 
relatively enduring 
predisposition to seek 
reengagement with a 
particular class of 
content over time 


Is likely to 
independently 
reengage content 

Has stored knowledge 
and stored value 

Is reflective about the 
content 

Is focused on their 
own questions 

Has positive feelings 


and a relatively 
enduring 
predisposition to 
reengage a particular 
class of content over 
time 


Independently 
reengages content 
Has stored knowledge 
and value 

Is reflective about the 
content 

Is likely to recognize 
others’ contributions 
to the discipline 
Self-regulates easily to 
reframe questions and 
seek answers 
Appreciates and may 
actively seek feedback 
Can persevere through 
frustration and 
challenge in order to 
meet goals 


e Has positive feelings 


Understanding how and why students become interested in STEM subjects can generate improved 
strategies for creating more students prepared for, and self-interested in, pursuing STEM careers, which is not only 
necessary (Aschbacher, Ing, & Tsai, 2014; Falk & Dierking, 2016), but also just and fair to all students and citizens. 
Interest development has been linked in past work to digital literacy and STEM educational pathways; e.g. in the 
work of Reynolds’ exploration in a 2-year-long after school computer club (Reynolds, 2008); the work of Shumei 
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Zhang & Callahan (2014) using Science Fiction Prototyping, through which students can create a relevant and 
authentic STEM-based item, real or imagined; and Ito’s (2010) research that resulted in the connected learning and 
HOMAGO models. However, digital inequality, whether it is derived from socio-economic, gender-based, or 
racially-based policies or beliefs, is a reality that must be addressed and corrected (Cooke, 2017; Riegle-Crumb, 
Moore, & Ramos-Wada, 2011). Generating interest in all students is the educational goal, and while some students 
seem to be predisposed to appreciation of STEM activities and thinking, many students simply do not like STEM 
subjects and their related concepts. STEM interest development and literacy, the social constructivist connection 
between STEM subject interest and involvement, and their implications for curricular development and 
enhancement of STEM-subject instruction and pedagogy, are represented in this study by the cross-pollination of 
Kuhlthau’s (1991) 6-stage, 3-domain, Information Search Process (ISP) and Hidi & Renninger’s (2006) Four 
Phases of Interest Development demonstrated later. 

Educational interventions can be useful in promoting learning, and STEM triggers can be utilized and 
combined in such interventions. Depending upon the target audience, lessons will probably employ situational 
triggers and/or appeals to mastery and individual interest, since those have been demonstrated to more frequently 
lead to STEM interest. The greatest challenge is the actual measurement of interest levels and changes. 


Importance of Identity and Expertise in Interest Development 

Interest is a complex blend of learner context, competing cognitive goals, cross-schema connections, affect, 
behavior, and desire. In her landmark study “The Case for Motivated Reasoning,” Kunda (1990) establishes that, 
“The motivated reasoning phenomena under review fall into two major categories: those in which the motive is to 
arrive at an accurate conclusion, whatever it may be, and those in which the motive is to arrive at a particular, 
directional conclusion” (Kunda, 1990, p. 480). This has profound implications for learning, since students must be 
convinced that either a subject and its associated activities and involvements should be engaged because it is socially 
appropriate and acceptable based on established standards, or because involvement is personally desirable due to a 
variety of social and inter-generalizable, contextual factors. Walsh & Tsurusaki (2017) note that, “Learners engage 
with social and material entities that mediate development and the transfer of knowledge, practices, and identities 
across an ecosystem and through time . . . forging connections between settings” (p. 6). It is those connections that 
drive engagement and interests, and potentially learning. One of the goals of triggering and/or maintaining STEM 
interest and involvement is to capitalize upon and expand student contexts and interests, incorporating the authentic, 
meaningful, relevant content considered most appealing to the students, making it normal and appropriate to 
envision working with future technology by reading and practicing: “Learners’ experiences within a particular 
environment are shaped by both their prior participation in particular communities as well as the structures, 
supports, norms, and expectations of the current environment” (Walsh & Tsurusaki, 2017, p. 36). Scaffolding, the 
raising of students’ abilities through mentorship, facilitation, etc., and helping students to achieve their zone of 
proximal development (the area between what a person believes she knows about a subject and what she actually 
knows, and the space in which mentorship is enabled) (Vygotsky & Cole, 1978; Reiser & Tabak, 2014), can be very 
supportive of enriching personal and group contexts by expanding and interconnecting schema through the 
development of expertise (Goldman, 2001). Connecting to context to motivate students is a significant piece of this 
puzzle, but certainly not the only one. 

The process to connect students to STEM-related disciplines, interests, and activities is not simply an 
academic exercise; it is an effort to inspire future generations to continue the innovative spirit that John F. Kennedy 
invoked when he insisted that Americans land on the moon by the end of the 1960s. It is the crucial work of forging, 
through STEM education, the next Steve Jobs, Mae Jemison, Neil deGrasse Tyson, or Marie Curie, of guaranteeing 
future generations the technological advances that postwar generations have enjoyed. Van Horne & Bell (2017) 
answer that call with social practice theory in an effort to, “. . . analyze how moments of interaction related to 
scientific practice advance (or constrain) learners’ trajectories toward becoming socially recognized and networked 
participants in disciplinary work” (Van Horne & Bell, 2017, p. 442). The more students identify with a discipline 
such as science or mathematics as a part of their identity, the deeper their potential involvement in related activities. 

Incorporating identity construction around content can account for potential student motivation to pursue 
STEM-related interests and activities, and there are ample opportunities within the theories for interventions 
throughout the process. In this manner, triggering becomes an essential element of learning and motivation. As Van 
Horne & Bell summarize, “. . . material, relational, and ideational identity resources and qualities of the learning 
environment mediate multiple entry points for different students” (p. 471). Recent STEM pedagogy links interest 
development, particularly for marginalized and stereotyped youth, to a social, interactional process that is “. . . often 
mediated by how students perceive the valued ways of knowing and being of a given practice or discipline in 
relation to the ways of knowing and being with which they already identify” (Pinkard, et al., 2017, p. 481). This 
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perception of a subject area like STEM can be enhanced through self-actualization by pursuing expertise in a 
particular STEM-related field. 

Identity construction is crucial to engagement in STEM subjects. As students gather and process seminal 
STEM-related experiences, informal (non-school related) elements become increasingly important. Maybe students 
should be afforded more opportunities to visit science-related sites, in-person, online, or both, during school hours 
(Subramaniam, et al, 2013). Legitimate Peripheral Participation (Lave & Wenger, 1991) suggests that as 
participants, in this case STEM students, become more involved in an activity or organization such as a Coding Club 
or a Conservation Team, they move towards the center of that activity and assume more ownership and 
responsibility in the organization and the activity. Sometimes “it takes a village” to bring scientific literacy to all 
students; for example, Ahn, et al’s (2018) project to increase awareness of science topics, fields, and information 
using public, neighborhood displays so young people became more engaged with STEM subjects. 

Expertise increases and more understanding becomes possible as context is built in a subject area. 
Background knowledge of content influences meaning construction through intertextuality, enabling students to 
more easily recognize similar topics, genres, and text features (Leinhardt & Young, 1996). Laypeople generally trust 
experts who are presented as such if they can establish a consensus concerning the data. Students are eager to seek 
and accept expert information, hence their willingness to accept the information in Google Answer Boxes as factual 
with little or no vetting (Miklosik & Dano, 2016). Uncertainty is important in a search (Chowdhury, Gibb, & 
Landoni, 2011); it is one of the main factors in enabling the user to make good decisions. Uncertainty decreases as 
the information seeking progresses until it virtually disappears at the end, but there is not uncertainty to eliminate if 
it never appears from the beginning (Chowdhury, et al., 2011). Replacing healthy uncertainty in favor of an answer 
box is a questionably valid school research option, since students, depending on their ages, are taught to use 
multiple, trusted sources to write papers when doing research; answer boxes are not always cogent or relevant. 

Students are much more inclined to achieve expertise if they can connect personally to the material, since 
they are already socially instructed to accept that speakers talk about what they know, and laypeople tend to accept 
expert consensus as factual or at least sensible (Goldman, 2001). Therefore, expertise plays a crucial role in 
motivation and identity construction, since people identify with what they know and believe, and are only able to 
scaffold on to what knowledge and learnings they already have (Kuhlthau, 1991; Reiser & Tabak, 2014). Students 
building expertise in a subject area, expanding context and deepening associations with specific disciplines like 
computing and engineering, are key to building STEM interest and involvement (Pinkard, et al, 2017; Van Horne & 
Bell, 2017). Additionally, there are specific storylines or narratives that are prevalent within societies and they can 
be utilized for learning new subjects (Pinkard, et al., 2017). The use of triggers to STEM education can not only 
enhance involvement and interest, it can also build context as it connects to already-existing narratives with which 
many students are familiar and comfortable (Kuhlthau, 1991; Reiser & Tabak, 2014). This approach is ideal for at- 
risk students who may not have as many inherent potential entry-points as other students with stronger support 
systems outside of the classroom; for example, narratives can spark interest in STEM learning activities for middle 
school girls (Pinkard, et al., 2017), a population long-neglected in STEM fields. 

Personal agency is critical for students, and hopefulness and pride result from its successful implementation 
(Ely, Ainley, & Pearce, 2013). The range of positive feelings associated with interest offered no surprises or 
unexpected results; kids are more likely to respond with positive affect concerning the stuff they like, and they are 
more likely to respond with negative affect concerning the stuff they do not like. However, identifying the specific 
emotions is less impactful on practitioners than simply knowing that positive and negative affect surrounding 
interest affects learning, and emotions combine with cognition, behavior, and ecology to form holistic information 
search procedures specific to those ecologies (Kuhlthau, 1991). 


STEM Interest Development and Information Search Processes 

In discussing information needs and desires, and the sense-making that should result from their pursuit, 
Case (2007) describes two poles of user approaches: Objective and Subjective. Objectivists view information as a 
reflection of a tangible reality, and the information search as a rational method to fill a specific need for information 
that is expected to somehow improve one’s life or situation, such as researching two seemingly equal job offers to 
determine which will be a better fit. The goal of the search is often cognitive gap-filling, such as finding that the 
commute to one of the jobs is shorter, making that job more lucrative and therefore better. Subjectivists represent 
information search much like Kuhlthau in her landmark 1991 model of the information search process (ISP) (see 
Figure 1). Subjectivists have a personal experience, enduring varying degrees of uncertainty and doubt through the 
first half of the ISP. Their goals range from simply filling an information gap, like finding out travel times and 
distances between locations, to reflecting an affective and behavioral process as well, such as deciding whether or 
not to get a divorce, to pursue a controversial medical treatment, or to go sky diving, in which there are existential 
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threats to be considered and anguished over. Subjectivists allow sense-making of a situation to act as powerfully and 
dominantly as a definite, measurable answer; there is not necessarily a right or wrong answer to whether one should 
sky dive or divorce one’s spouse— those decisions are based on a combination of influences from all three domains 
Kuhlthau (1991) represents in her ISP. 

Carol Kuhlthau is among the most cited and discussed researchers in the information behavior field (Todd, 
2003). Innovating beyond the cognitive-affective models and theories that preceded her, Kuhlthau insists that a 
model of a user’s sense-making process during the engagement in the ISP, “. . . ought to incorporate three realms of 
activity: physical, actual actions taken; affective feelings experienced; and cognitive, thoughts concerning both 
process and content” (Kuhlthau, 1991, p. 362). Objectivists are binary—either the information is satisfactorily found 
through cognitive methods or it is not—while Subjectivists allow for a range of responses across multiple domains 
(Case, 2007). The theories to be discussed here are Subjectivist. This approach will enable students’ emotional and 
behavioral responses to stimuli to be considered a part of the information search and enable another avenue for 
strategic interventions, and provide a lens for a social constructivist view of the process. Students often settle for 
sense-making over knowing, such as seeing and acknowledging the myriad changes they undergo during 
adolescence but not actually understanding the ways in which all of those factors gel together (or do not in the case 
of dysfunction) to eventually create an adult; or watching the night sky and having a sense of the cosmic forces at 
play without acutely understanding Einsteinian physics, choosing awe and wonder as adequate sense-making over 
definitive astronomical and gravitational explanations. 


Model of the Information Search Process 


Initiation Selection Exploration Formulation Collection Presentation Assessment 
Feelings Uncertainty Optimism Confusion Clarity Sense of direction/ Satisfaction or Sense of 
( Affective) Frustration Confidence Disappointment accomplish- 
Doubt ment 
Thoughts vague. ——_ focused Increased 
oe: oe ote 
(Cognitive) : self 
increased interest awareness 


Actions seeking relevant information seeking pertinent information 


(Physical) Exploring Documenting 


Figure I. Kuhlthau’s (1991) 6-stage, 3-domain ISP Model 


The reason Kuhlthau’s work is so important in this space is her acute understanding of the roles that behavior and 
emotions play in information search. Kuhlthau accepts that searchers feel human emotions and experience human 
behaviors. They are frequently not the bold information-adventurers educators strive to create, opting for safety, 
ease, and comfort over completeness and thoroughness, choosing the known over the unknown. This is supported by 
the epistemology of Scaffolding theory (Reiser & Tabak, 2014) that asserts knowledge is only usable as an 
extension of previous knowledge, added-on to the original knowledge and integrated based on context and pre- 
existing schema. Kuhlthau recognizes that and boldly makes it one of the features of her model: 

Since people have a limited capacity for assimilating new information, they purposefully construct meaning 

by selectively attending to that which connects with what they already know. The active process of forming 

meaning from information is the task of the user in the ISP. An information search is a process of 
construction which involves the whole experience of the person, feelings as well as thoughts and actions. 

(Kuhlthau, 1991, p. 362) 

Activating and building rich contexts for interpreting information using familiar concepts and constructs is 
the user’s natural tendency, so the richer and more diverse the students’ contexts, the more potential connections can 
be made. Additionally, Kuhlthau’s foundational incorporation of personal construct theory, describing the affective 
experience that accompanies meaning construction, into the ISP model represents a leap in information search 
understanding as the fields of information processing, education, and psychology blend in the final representation. 
The implication for students is powerful: if a particular student is not cognitively connecting, enjoying, or 
appreciating a STEM subject, field, or lesson, there is a model to express that cognitive, affective and/or behavioral 
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experience, target that moment of disconnection in the process, and potentially influence it with a STEM-based 
intervention. Kuhlthau (1991) provides the beginning of a theoretical foundation for such interventions. 

When students search for information, they engage in “forming new constructs and altering those 
previously held” (Kuhlthau, 1989). As students realize the problem they face, the information they lack to complete 
their assignment, they encounter a“. . . recognized anomalous state of knowledge (ASK), which, further modified 
by linguistic and pragmatic considerations, becomes a request put to the IR [Information Retrieval] system” (Belkin, 
1980, p. 135). Thus, a communication is established between the technology that retrieves the information and the 
user who requests the information (Belkin, 1980). Information, content, and experiences translated internally into 
language is scaffolded upon previous, related schema, enabling completion of increasingly complex tasks (Reiser & 
Tabak, 2014). With a tutor, facilitator, mentor, teacher, or other trained collaborator, students achieve their zone of 
proximal development which enhances their abilities (Vygotsky & Cole, 1978; Reiser & Tabak, 2014). Students 
must learn to navigate difficult terrain in the technology-rich, answer box era, acting and being acted upon by forces 
both inside of their personal experience and from external, societal forces (Schutz & Luckmann, 1973). 

As students encounter gaps in their “stock of knowledge” (Schutz & Luckmann, 1973, p. 100), learning and 
attempt to fill those gaps becomes the primary goal. However, there are limiting factors to the stock of knowledge 
such as situation, spatiality, temporality, and social arrangements (Schutz & Luckmann, 1973). Temporality 
suggests that time is a factor in information behavior concerning searches, and that the search and the searcher are 
interrelated (Beheshti, et al, 2015), creating space for both chronos, or chronological time, and kairos, time defined 
by shared experience, social constructivist time. Since searchers can only assimilate a certain, finite amount of 
information in a search session, they purposefully construct meaning by choosing those slices of information they 
know and connecting new information to them (Kuhlthau, 1991). 

Kuhlthau continues to address the Affective element as having a significant effect on cognition: “Affective 
aspects, such as attitude, stance, and motivation, may influence specificity capability and relevance judgments as 
much as cognitive aspects, such as personal knowledge, and information content” (1991, p. 363). Personality traits 
are an active element of Kuhlthau’s Affective medium: “As personality forms an inclination towards certain 
characteristic reactions in any given situation, personality traits are likely to influence attitudes and behavior in an 
information-seeking context” (Halder, Roy, & Chakraborty, 2010, p. 43), creating a multi-domain effect and 
exemplifying the interrelatedness of the three domains. 


Cross-Pollination of Kuhlthau’s ISP (1991) and Hidi & Renninger’s Interest Model (2006) 
This paper introduces a new element to interest development research: a cross-pollination of Kuhlthau’s 


(1991) 6-stage, 3-domain ISP Model with Hidi & Renninger’s (2006) Four Phases of Interest Development. The 
hypothesized connections are demonstrated in the model below (see Figure 3): 
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Figure 3. Hypothesized Model of Cross-pollination of Kuhlthau’s (1991) 6-stage, 3-domain ISP Model with Hidi & 
Renninger’s (2006) Four Phases of Interest Development 


In Figure 3, as students collaboratively search, retrieve, and utilize information, increasing relatedness 
between themselves and the content, the initial uncertainty and vague sense of a new content area is reflective of 
triggered situational interest, the first interest phase. During the /nitiation phase of the ISP, new information is 
assimilated in a series of phases, scaffolding upon existing context (Kuhlthau, 1991; Reiser & Tabak, 2014). Short- 
term changes in affect and cognition lead to mixed results; students may or may not persevere and develop 
relatedness and begin to develop competence and they may need collaborative support to “stay in the game” (Hidi & 
Renninger, 2006). If they stay engaged in the material and develop optimism about their potential for success, 
working though their vagueness concerning this still-new topic, possibly with their classmates (Kuhlthau, 1991), 
students can develop maintained, situational interest in which they develop increased focus and attention on the 
information seeking and retrieving task, increasing scaffolded content information, sharing and confirming 
relevance with other students, and generally feeling positive about the project (Hidi & Renninger, 2006). As 
questions become requests, reflecting an active cognitive engagement with the content (Belkin, 1980), students 
approach Formulation, the “. . . turning point of the ISP” (Kuhlthau, 1991, p. 367), in which affect and cognition 
grow from frustration and confusion to clarity and greater focus. This point is comparable in the ISP journey to 
emerging individual interest in interest development, during which students begin to seek information for 
themselves and their own budding interests, not just those related to the school project (Renninger & Hidi, 2016). 
They have been successfully triggered and are on their way to well-developed, individual interest, featuring personal 
connections to the content that transcend the immediate context (Renninger & Hidi, 2016). This phase is represented 
in the ISP by Collection, “. .. when interaction between the user and the information system functions most 
effectively and efficiently” (Kuhlthau, 1991, p. 368). The seamless transition described represents ideal conditions; 
there are many issues and distractors such as seductive details and technology lapses (Wang & Adescope, 2016) that 
can interfere with this process. The scaffolding that has been dynamically expanding throughout this process has 
been serving students well, offering real-world problem-solving abilities, reducing frustration while increasing 
interest potential, offering multiple foci for future exploration, and thereby enabling reflection and reflexivity 
(Reiser & Tabak, 2014). It could be suggested that it is this lack of cognitive load that enables the positive feelings 
that often accompany Collection. This connection between the work of Hidi & Renninger (2006) / Renninger & Hidi 
(2016) and Kuhlthau (1991) will be explored and clarified further in future research. 
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Social Constructivism and School Librarians 

Students share information, view of triggers and their transference to STEM interest through a 
constructivist lens, more specifically Piaget’s radical constructivism and Vygotsky’s sociocultural constructivism. 
Steffe & Kieren (1994) offer a particularly lucid explanation of radical constructivism by pitting it against 
Chomskyian innatism. In this light, Chomsky argues that all subject structure comes from within individual 
cognition, while Piaget posits that cognitive structures continue to be constructed and deconstructed throughout 
childhood and possibly adulthood. This offers STEM teachers a difficult path, since in a constructivist classroom, 
there are no pregiven, prescribed ends toward which this construction strives, and assessment and proof of learning 
become paramount. If a constructivist learning environment is assumed, then educators should be required to offer 
easily apprehensible models of content-based concepts, because the outcomes processes students use to get to the 
answer and the outcomes themselves may differ (Cobb, Yackel & Wood, 1992). In Piagetian terms, “. . . students’ 
informal mathematical activities constitute a starting point from which the teacher can guide their problem-solving 
efforts and thus facilitate their acculturation into the mathematical ways of knowing of wider society” (Cobb, Yackel 
& Wood, 1992, p. 13). Since radical constructivism is highlighted by students who individually create and re-create 
their identity from initial assimilatory structures, teaching for conceptual understanding takes precedence over 
content-based or unauthentic activities (Fosnot, 1992). 

Vygotsky’s sociocultural constructivism differs from Piaget’s radical constructivism in that, for Vygotsky, 
social experience shapes the ways of thinking and interpreting the world. Even individual cognition is a social 
activity, especially when it involves language and other semiotics (Jaramillo, 1996); every element of human 
experience that involves interaction with people and/or the environment is social. Sociocultural theories tend to 
focus upon how individuals learn and the cultural influences on those processes, while constructivist theories 
describe how epistemologies function in learning environments in general (Dolberry, 2015). Furthermore, since 
learning extends over multiple years, educators can consider and re-consider how topics are presented at each grade 
level each year, building on prior understanding and supporting increasingly complex concepts (Next generation, 
2013). The only alternative to a constructivist approach to information and knowledge dissemination is a 
transmission approach (Kuhlthau, Caspari, & Maniotes, 2015), the current equivalent of “chalk and talk,” a method 
that may be effective in some university classrooms but almost never in middle school classrooms. Vygotsky’s 
theories of higher-order thinking call for authentic learning, so it is incumbent upon school librarians to provide a 
constructivist learning environment that empowers students to explore content and processes drawn from their own 
experience and meaningful to them (Kuhlthau, Caspari, & Maniotes, 2015). Constructivist learning can be perceived 
by students as a very complex situation, and it can at times be overwhelming to “buy in” to an unfamiliar 
methodology (Perkins, 1992). Also, school librarians and subject area teachers are faced with particular challenges 
in maintaining a constructivist learning environment, especially when it comes to assessment: Is using only one 
assessor an unbiased approach? Can a question ever have less than several correct answers? How are skills, not 
knowledge, assessed? (Jonassen, 1992). 

As much as educators may want to offer choices to their students and facilitate their constructionist journey 
through them, curricula, instructional strategies, and objectives are often imposed on students and staff from outside. 
However, this non-constructivist approach cheats students of opportunities for self-efficacy and identity construction 
(Winn, 1992). The work of Piaget supports construction of identity and self-efficacy through interaction with the 
environment, growing, developing, and evolving within it, reinventing and reorganizing information and 
understanding along the way (Fosnot, 1992). As students progress through Piaget’s stages—from concrete 
understandings, to symbolic representations of those understandings, to abstract models—they learn structure, order, 
and reflection. Then, through interactions with others, their understandings change again (Fosnot, 1992). This is the 
process through which students come to own their research and their skills, fostering a growing curiosity that serves 
students well (Kuhlthau, Caspari, & Maniotes, 2015). All information search has a social constructivist element in 
general insofar as the information individuals encounter is embedded within the culture and its norms, language, 
economic opportunities, and regional idiosyncrasies (Subramaniam, et al, 2015). Therefore, even without calling it 
constructivist, the elements of constructivist education emerge. 


Guided Inquiry 

Supporting a dynamic social constructivist STEM classroom or school library requires pedagogies and 
practices that support both abstract and concrete understandings of content and processes that, through scaffolding, 
become usable learning (Reiser & Tabak, 2014; Kuhlthau, 1991). Employing guided inquiry is one effective method 
of having students collaborate in a social constructivist environment while progressing through the stages of 
Kuhlthau’s (1991) 6-stage, 3-domain ISP (Kuhlthau et al., 2015) In Guided Inquiry (Kuhlthau, Maniotes, & Caspari, 
2015), the goal is deep understanding of relevant content while constructing and developing content-area literacy 
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and social skills, leading to constructivist, sociotechnical, “third space” (Kuhlthau, Maniotes, & Caspari, 2015, p. 4) 
learning environments. Sociotechnical systems are “places” in which humans and/or organizations interact with 
technology (Hu, Mostashari, & Xie, 2010). It is connected to Kuhlthau’s (1991) information search process and 
therefore particularly relevant in this study. It is intended to be collaborative, engaging, reflective, and iterative, as 
students explore new ways to solve problems, answer questions, and create artifacts. Motivation, interest, relevance, 
and affect are primary factors in engaging in guided inquiry learning. Figure 3 is an updated version of the original 
ISP in a guided inquiry setting. Students use multiple information sources, researching both individually and as part 
of an Inquiry Community, a team of researchers/presenters. The end products are almost always concrete artifacts 
based upon the process the students experienced. Even virtual creations are concrete insofar as they can be 
transformed into an object if needed. These artifacts can shed light upon both the process and the learning that 
occurred (Kuhlthau et al., 2015). Guided inquiry is also closely connected to motivation and interest, and analysis of 
both the research process (with videotaping and field notes) and the artifacts produced in the planned intervention 
should inform the research concerning learning, literacy, and the potential cross-pollination of Kuhlthau’s ISP with 
Hidi & Renninger’s (2006) interest development model. Since all of this action is occurring in a group setting, not in 
isolation, a project-based pedagogy is also needed. 

Educators may be partially responsible for not engaging students enough with real-life experiences 
communicated in a style that both students and educators can relate to (Pauw, et al, 2015). Even if students build 
more science-related context and experience, they still need impetus to make connections to previous funds of 
knowledge which are not always activated. Many learners have the skills to make meaningful connections but 
choose not to. However, social media platforms are a bridge to such connections, and online groups should be 
organized around science learning to enhance the possibility of critical connections (Mills, et al, 2018). Social media 
can help students further craft their identities as future scientists by discussing and sharing their experiences, 
affirming their choices with their peers (Subramaniam, et al, 2012). Achieving these goals takes dedication by both 
students and staff, and it helps if they both like the material, and equally importantly, if staff stay current in their 
fields (Mills, et al, 2018). 


Guided Inquiry Design Process /\ ae 
\/ 
Shore 


” Arrows indicate social constructivist 
pen 


collaboration opportunities 


Figure 4. Guided Inquiry Design Process (Kuhlthau, Maniotes & Caspari, 2015) 


Guided inquiry is both grounded in best practice and research and crucial for successful living and 
transition to the workplace (Kuhlthau, Maniotes, and Caspari, 2012). Guided inquiry is authentic to students because 
they work with peers and with chosen, not assigned, subjects, or at least chosen sub-topics within a curricular theme. 
Technology and its ability to simultaneously create and share plays an important role in successful critical skill 
building in students, the ultimate goal of guided inquiry design. 

In order to map out the information search process using guided design theory, Kuhlthau, Maniotes, and 
Caspari (2012) designed a new 8-step ISP model to augment the previous 6-step ISP cited above (see Figure 4). The 
newer model includes opportunities for students to practice the guided inquiry behaviors of analysis and sharing. 
The added arrows represent the times at which students collaborate, discuss, debate, explore, question, and assert, all 
important steps to knowledge, socially constructing while learning. In a sociotechnical view, the “third space” that 
Kuhlthau describes, demonstrated in Figure 3 above, is a significant part of the knowledge. 
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Conclusion 

Every middle schooler is entitled to the tools she needs to succeed in an uncertain future. STEM interest, 
and the teachers and school librarians who promote STEM, need to be a part of that future. They do not merely 
create an environment in which higher standardized test scores become more likely (Lance & Hofschire, 2012). The 
school librarian may be the only individual in the building “. . . tasked with building and maintaining a collection of 
diverse, high quality, current resources that support curriculum, complement adopted texts, enable professional 
learning, and pique student interest” (Mardis, 2014, p. 250). AASL defines the five primary roles of school 
librarians: information specialist, instructional partner, teacher, program administrator, and leader (American 
Association of School Librarians, 2014). No one else in the building is capable of this range of capabilities. 
Additionally, and probably in response to a growing movement, they describe librarians’ possibilities of creating 
semi-formal learning environments in school libraries, excellent choices for librarian collaboration since they can 
offer more media, more guidance, and more interest-driven topics and themes. 

Without guidance, students often approach the process as a simple collecting and presenting assignment 
that leads to copying and pasting with little real learning. With guidance, students are able to concentrate on 
constructing new knowledge in the stages of the inquiry process to gain personal understanding and transferable 
skills. Students’ feelings play an important part in the constructive process of inquiry that indicates a zone of 
intervention for teachers and librarians. For example, students get frustrated in the exploration stage of inquiry and 
need encouragement to take time to read and reflect and guidance in making sense of information that does not fit 
together smoothly. Guided Inquiry provides essential intervention at critical points in the inquiry process that fosters 
deep personal learning (Kuhlthau, 2010). Guided Inquiry prioritizes and defines that space for students and staff 
alike by providing opportunities for engagement and reflection (Kuhlthau, Caspari, & Maniotes, 2015), especially 
since students often mis-assess their searching skill and then wrestle with the cognitive, affective, and behavioral 
ups and downs predicted by Kuhlthau (St. Jean, et al, 2015). Self-reflection helps students and teachers during this 
time, and both benefit from activities like log-keeping (Harada, 2016). Guided inquiry is a response to the increasing 
need for school transformation; the Internet alone does not create a 21‘ Century learning space (Kuhlthau, Caspari, 
& Maniotes, 2015). Facts run the risk of becoming fluid in this era of rapid technological change. The Internet 
provides unparalleled access, and offers a voice to all who want to have one, but, as Kuhlthau (2015) warns, “. . . 
this also produces an abundance of misinformation and misunderstandings, intended or not. Questions arise of what 
is accurate, reliable, important and wise. There is confusion between what is enduring and what is ephemeral” (p. 2). 
The Internet makes fake news possible. Kuhlthau accurately predicts that with voice comes responsibility, and 
people approach their newfound voice with different motives: some to inform and assist, but also some to sell, some 
to prey, and some to ignobly persuade. Middle schoolers often have a difficult time distinguishing online fact from 
fiction, conflating the power of a story with its truth value; advertisers and predators know that (Gretter, Yadav, & 
Gleason, 2017). It is possible that no amount of media literacy education can counter the effects of disinformation 
(boyd, 2018), though Hobbs (2018) reminds boyd that media literacy is more than simply deconstructing 
communication; it involves reflection, meaning-making, and authentic action. These processes often involve 
triggering interest and/or engagement, and they often occur in a social constructivist learning environment. 
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Introduction 


Even as curricular content has become increasingly standardized and teachers have decreasing choices 
about what material they can present, the popularity of science fiction (SF) has returned to the STEM (Science, 
Technology, Engineering, and Mathematics) classroom. Researchers are exploring this potent resource through the 
lens of interest development. School librarians, in a welcoming environment for all students, offering project-based 
curricula driven by guided inquiry and information searching skills, is a potentially effective fulcrum around which 
STEM subject learning revolves. 

STEM education remains one of the hottest topics in educational circles, and countless opinions are being 
forwarded nationwide about how to best present this material and motivate all students to participate more fully in 
STEM education and STEM-related activities (Herschbach, 2011). Since the 1950s, educators have touted the use of 
science fiction (SF) literature both as a motivational teaching tool and as a method to integrate and lessen the anxiety 
level of some students who fear what they do not yet understand. While some students seem to be predisposed to 
appreciation of STEM activities and thinking, many students simply do not like and/or are intimidated by STEM 
subjects and their related concepts. Working with SF does have a generally positive effect on student attitudes 
toward STEM subjects (De Lepe, et al, 2015) and it shows great promise for STEM interest activation and 
maintenance. 


Why SF? 

One of the strong connections between SF, STEM, and career choice is the reality that SF has been 
dominated through the years as much or more by scientists as by literati. This is one unfortunate but genuine 
explanation of the gender imbalance in published SF up until this time. An unfortunate misogyny and patriarchy 
have dominated both authorship and subject matter of SF (Levy, 1999). Generally, gender has the most situational 
effect on topic interest (Ainley, Hillman, & Hidi, 2002), and situational interest is triggered much more than 
individual interest. Topic interest also affects persistence, and to a lesser degree, affective response. Girls are more 
likely than boys to explore lower-interest titles. When topic interest triggering influences affective responses, 
reading persistence, and ultimately learning. Lower interest materials require more persistence, something girls have 
more than boys in those situations (Ainley, Hillman, & Hidi, 2002). This reading persistence is something that 
should be fostered in all students regardless of gender. These levels should be addressed to ensure that STEM career 
possibilities are available to the greatest number of students. 

In Science Fiction by Scientists (2017), editor Michael Brotherton professes both his love of science and the 
clear connection between science and scientists by introducing and compiling stories by active researchers, 
“working at universities, medical schools, and space agencies . . . [and including] astronomers and physicists . . . 
neuroscientists, computer scientists, and rocket scientists” (Brotherton, 2017, p. vi). SF is popular with engineers 
and scientists because it “. .. connects with the human spirit of adventure and imagination” (Shumei Zhang & 
Callaghan, 2014, p. 353) and empowers designers to innovate. Scientists like SF so much that some of them feel 
compelled to use their STEM-subject related experience and interest to write SF and disseminate and share their 
appreciation for the role that SF played and continues to play in their lives and careers. Barry Luokkala (2014) also 
begins his book Exploring Science through Science Fiction by professing his early love of science and SF, but after 
realizing that SF can be used to teach Science, he begins to incorporate SF film clips and television episode excerpts 
into his classes to use as a springboard for scientific learning. His seven areas frequently found in SF (Luokkala, 
2014) offer a manageable framework to examine student interests and find potential entry points into student 
contexts, with augmented subject suggestions: 
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Nature of space and time (Astronomy, Physics, History of Science, Gravitation) 
Composition of the Universe (Geology, Matter, Light, Energy) 

Machine consciousness (HCI, Artificial Intelligence) 

Aliens (Anatomy, Biology, Zoology) 

The meaning of being human (Scientific method/Taxonomy, Ethics, Values) 

Solving future problems (Hydrology, Meteorology, Technology) 

What does the future hold? (Computer Science, Genetics, Astronautics) (Luokkala, 2014) 


The seven SF realms provide a structure with which to examine practically any STEM-related subject. Middle 
schoolers learning about the elements might deal with the Universe’s composition by reading and watching clips of 
The Martian (Scott, et al, 2016) and examining how the protagonist uses his natural environment to survive, while 
high schoolers might read and watch The Bicentennial Man (2008) to prepare for a discussion of ethics and human- 
computer interaction (HCI). Luokkala (2014) also offers offbeat, STEM-related issues and questions that will appeal 
to many students, such as the potential effects of launching students into space from a giant gun (Physics, Geometry, 
Mathematics), possible first contact scenarios between humans and extraterrestrials (Anatomy, Biology, Zoology), 
and future solutions to feed the hungry (Agriculture, Technology, Genetics). 

As a middle school librarian who introduced SF into a semester-length 7 and 8" grade Robotics class, I 
have witnessed the light in students’ eyes when they watch a clip of Star Trek: The Next Generation’s Commander 
Data (Stewart et al., 2013) display his superhuman strength, intellect, and relative immortality, and I have 
experienced my students’ passion for popular books and movies such as The Hunger Games (Collins, 2008) and The 
Maze Runner (Dashner, 2014), both of which offer multiple entry points into STEM-related fields. This is a rich 
area, ripe with possibilities for inspiring the next generation of scientists. Practicing scientists engage in literacy for 
a range of purposes, such as explaining their findings, conducting research, connecting to the work of others, or 
sharing ideas with the public. The increasing diversity of available texts and the accumulation of scientific evidence 
requires a scientifically literate person to be able to critically interpret an author’s stance and purpose, distinguish 
relevant from irrelevant evidence, and use complex information to make decisions (Goss, Min, & Strohl, 2016). 

The current plethora of popular examples of SF with adolescent characters, both young adult (YA) and 
adult, indicate student interests, and this should be incorporated into STEM-related lessons to enhance interest in the 
technology those characters use. The choice of a range of speculative fiction (encompassing SF, fantasy and other 
imaginative ideas) like Harry Potter (Rowling, 1998) and The Hunger Games (Collins, 2008), or uchronic literature 
like The Man in the High Castle (Dick, 1962), a current Amazon television series, is a logical one for adolescents, 
considering the connections between SF and the Bildungsroman, or what middle and high school teachers call the 
“coming-of-age” novel. It is not surprising that many YA series have spawned popular film adaptations. The 
alienation characters experience in both novels, the way characters feel out of place in their given environments, is a 
very common theme for adolescents, i.e. Harry is a potential hero at Hogwarts, a fantasy construct, yet he is shunned 
and misunderstood at home (Rowling, 1998); all of the characters in Dick’s novel are existentially in the wrong 
reality and living a life that never should have existed (Dick, 1962). Buckley (1974) describes the typical 
Bildungsroman plot as follows: a child gifted in some way is constrained and alienated at home and must leave, and 
her education in the ways of the world or in the methods of achieving success become equally or more important 
than school lessons. At the end of the journey, she has been exalted and debased, loved and loathed; she ultimately 
loses her adolescence and begins her adult journey, sadder and wiser than when she began. That is Harry Potter’s 
journey throughout Rowling’s series, and since his story appeals to a wide audience, his use of technology, such as 
flying during Quiddich, could be a catalyst for a lesson on forces, physics, gravity, mathematics, medicine (if he 
falls), or any number of STEM subjects. It is also Katniss Everdeen’s journey in The Hunger Games (2008), a novel 
that invites lessons on genetics, light, sound, flammable and inflammable materials, weaponry, and an actual 
example of the media’s “fake news” through the representation of the corrupt administration of President Snow 
(Collins, 2008). The Bildungsroman is the drama of the gifted child (Miller, 1990). 

The “bright kid” is one of the most memorable and enduring of the standard characters who comprise a SF 
story, mainly because many SF fans and authors begin their lives, or see themselves as, those same bright kids, often 
without the ability to flower and bloom due to societal, familial, cultural, gender-related, or other obstacles, like 
Harry Potter locked in his closet at home, only able to become his true and fully-actualized self at Hogwarts (Disch, 
2005; Rowling, 1998). Those kids often feel out of place and alienated during adolescence, as most people do during 
rapid industrialization, global wars, deterioration of the cities, and pollution of the environment (Toffler, 1990), all 
common themes in SF. For example, Asimov’s Foundation (1951) describes the capital planet of Trantor as so 
overdeveloped that the only open land on the entire planet is the Emperor’s palace grounds; Haldeman’s The 
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Forever War (Haldeman, 2009), published at the end of the Vietnam War era, depicts a never-ending war across 
time; Harrison’s Make Room! Make Room! (Harrison, 2008), the basis for the popular and still-discussed film 
Soylent Green (Fleischer et al., 1991), is about a world so overpopulated that cities have become slums, drug use is 
rampant, the environment is a dumping ground, and food is the only important currency that matters to the teeming 
masses. The “bright kid” is a powerful archetype that should be much more inclusive and appealing; although it is 
assumed ethically that no one wants to put undue pressure on students, expertise is best developed with a model that 
feels authentic and a personally-significant reason to strive for it (Goldman, 2001). 

In describing the goals of his 2014 work Exploring Science through Science Fiction, Barry Luokkala 
(2014) assumes the connection of SF to science education, promising in the book to use SF as a starting point and 
anticipatory set for the exploration of popular science topics. Boiero de Angelo, et al (2008) describe the use of SF 
with English Language Learners (ELL) with enthusiasm, clearly indicating the potential connection to future STEM- 
related activities and careers by providing access to STEM subjects in a pleasant manner. Though their ELL students 
expressed surprise at first by the use of SF to teach English, they soon realized the pleasures and benefits: “At the 
end of the year, however, they felt that reading a story had helped them to access texts other than those specific to 
their careers, and manifested their interest in reading other stories in English” (Boiero de Angelo, et al, 2008, para. 
20). In a similar vein, SF and fantasy (SFF) could assist significantly in the development of science-specific 
vocabulary, “. . . thereby serving as a bridge from general literacy to scientific literacy” (Rolls & Rodgers, 2017, p. 
54). The use of previously-learned biological concepts in the classroom can be the foundation from which students 
study future directions in SF, offering an “. . . excellent time to develop student awareness of interrelationships 
among biological concepts” (Marks, 1988, p. 277). Scientific literacy is a key factor in STEM success; scientific 
literacy demands a new level of critical interpretation of research sources (Goss, Min, & Strohl, 2016). Students 
often enjoy producing an artifact at the end of a SF-related STEM subject unit, such as an original short story, poem, 
or film dramatization of a story. The use of technology to learn about technology is both pedagogically sound and 
important for the development of future productive technology use. The story of Neopets represents an excellent 
allegory of the power of digital artifact production. Owned by major media corporation Viacom, controllers of 
children’s network Nickelodeon, Robinson & Horst (2010) found Neopets nonetheless to be an engaging gaming 
site due to its adaptability to individual interests. For example, some participants simply liked the games, while 
others’ interests were drawn to the creative potential of the platform, while still others enjoyed socializing with other 
Neopets users and sharing their experience in a widening social constructivist environment. This variety offers 
multiple entry points for diverse audiences. 


The Use of SF in the Classroom 

SF curricular ideas and pedagogies appear in many forms. Barra (1988) allows students to create stories 
around chemical equations, formulae, and combinations, hoping to stimulate interest in chemistry specifically and 
STEM subjects in general. Brake & Thornton (2003) adopt a Dewey-like approach to science education, describing 
scientific literacy as a right and a requirement for every citizen of a democratic society. They assert that SF is an 
effective medium through which to examine the relationships between science, technology and society, not only 
inspiring students to create new ideas and technologies, but to also popularize and disseminate scientific ideas. 
Czerneda (2006) also notes that SF develops literacies, but she additionally observes that, “We are living in a world 
that seems science fictional, and SF readers have the advantage of knowing the terrain” (p. 39). With today’s rapid 
technological change and cries of fake news coming from every corner, this may be truer now than ever. Far from 
just making observations about SF and science, this area features specific curricular suggestions as well. 

SF continues to be a catalyst for STEM learning as it struggles to become a more common feature of STEM 
curricula. Burns (1994) offers the use of Jules Verne’s work to teach density and chemical decomposition with the 
intention of creating an environment in which students can discover the goals and objectives of lessons without 
having to be told, thereby enabling students to adopt more ownership of and personal attention to activities related to 
scientific inquiry. Sprague & Cotturone (2003) used The Science of Star Wars to teach physics and found their 
students amazed to discover that they knew more about science than they thought due to SF media exposure, as well 
as impressed by the actual complexity of concepts like artificial gravity and space travel that are often taken for 
granted in SF. Freudenrich (2000) asserts that a powerful approach to science teaching is the establishment of a 
contextual framework for learning, and that SF provides a flexible and accessible introduction to fields such as 
physics, astronomy, and biology. Freudenrich also argues, like many of the researchers who attempt the integration 
of SF and STEM, that SF media “. . . readily capture the interest of students, even those who are not SF enthusiasts” 
(Freudenrich, 2000, p. 45) and that both students and parents enthusiastically embrace this approach. Brake & 
Thornton (2003) indicate that SF is a cultural phenomenon than encourages scientific creativity and imagination, 
and it can be used not only to teach specific scientific concepts, but also to analyze “. . . the relationship between 
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science, technology, and society, both as an inspirational source guiding the direction of scientific development and 
as a way of popularizing and disseminating scientific ideas” (Brake & Thornton, 2003, p. 32). However, despite the 
repeated examples of successful implementation of SF into the STEM classroom, the concept never had the 
opportunity to “go viral” and achieve implementation on a state-wide or national level. 

In the last 10 years, the incorporation of SF into the STEM classroom has still remained in the realm of the 
practitioner, thriving underground but seeing very little of the light of broad implementation due to lack of research 
support. Studies published since 2010 discuss the use of specific works that have already gained worldwide 
recognition. Smyth & Waid (2010) present mathematics lessons designed for modern classics A Wrinkle in Time 
(L’Engle, 2007) and Harry Potter and the Sorceror’s Stone (Rowling, 1998). However, these ideas have largely 
thrived in limited environments. 

Though name recognition of series such as Harry Potter (Rowling, 1998), The Hunger Games (Collins, 
2008), and The Maze Runner (Dashner, 2014) can motivate student interest, not all SF encounters in STEM-related 
classes must be linked to specific works. Engel & Schmidt (2004) describe a creative and dynamic solution to 
integrate into mathematics lessons: The Galactic Spaceship Tour Challenge. Students are broken into groups and 
given a scenario in which they are the owners of a spaceship travel company and must calculate the miles, fuel costs, 
and other expenses involved in transportation to different star systems in order to charge fair but profitable prices 
and maintain their business. Engel & Schmidt (2004) maintain that solving individual problems may reinforce basic 
mathematical skills, but true development and advancement requires larger, more encompassing projects that 
actively engage students and involve them in social problem solving as well as calculation. Kay & Golden (1991) 
use a similar method, asking her students to imagine they are on an alien world and describe things like pets, jobs 
foods, etc. on the new world. Freudenrich (2000) establishes context with a SF film, then addresses scientific forces 
like gravity and acceleration shown in the film, while Dubeck, et al (1993) makes science relevant by examining 
concepts in SF films and discussing their potential for actually occurring, leading to discussions on scientific 
concepts employed in the examples. One particularly compelling idea is to use Star Wars radio serials to enhance 
literacy education (Davis, 2016). 

Some researchers have been more ambitious than simply suggesting one unit of study, using literature 
across the curriculum to promote the power of SF narratives to teach and encourage students to develop research, 
creativity, critical thinking, debating, and decision-making skills. Goldbort (1991) uses Mary Shelley’s Frankenstein 
and other literary works with similar themes, to explore some ethical and scientific considerations of such ideas as 
cloning, life-prolongation, and genetic engineering, while Burns (1994) suggests Jules Verne’s work for 
brainstorming on how to deconstruct scientific processes. Boblick (1991) digs up old movie serials from the 1940s 
and 1950s to discuss issues of propulsion, force, and gravity, whereas Smyth & Waid (2010) present ideas for use 
with many middle school-level SF books like Flatland and Harry Potter and the Sorceror’s Stone. 

Perhaps the most compelling idea for an SF literature project uses Susan Collins’ The Hunger Games 
trilogy, the most popular SF series of this generation; by 2012 over 50 million copies of the 3 books had been sold 
(Scholastic, 2012). Cook, Keller, & Myers (2014) offer a unit on genetic engineering featuring The Hunger Games 
(Collins, 2008), complete with tie-ins to national standards and lessons on both scientific literacy and ethical 
sensibility. Citing Next Generation and Common Core Science Standards, Cook, Keller & Myers (2014) raise real- 
world ethical and moral questions as they explore the science of genetic engineering, a practice that has existed for 
many years, in contrast to its use in The Hunger Games. Burton, Goldsmith, & Mattei (2018) may as well be 
describing middle schoolers when they assert that “. . . any field that involves practice requires not only technical 
proficiency of its practitioners but also ethical proficiency, as manifest not only in a command of the relevant 
knowledge but also the inclination and ability to let that knowledge take precedence over laziness or self-interest” 
(p. 58). The idea of ethics education as an element of media education and use is a potent and cogent one. Fiction 
mirrors life and can therefore represent a form of verisimilitude to which students can relate: “Fiction allows 
educators to reframe recognizable human situations and problems in terms of unfamiliar settings and technology” 
(Burton, Goldsmith, & Mattei, 2018, p. 60). The use of SF over other genres is supported: “Science fiction thus 
permits a curricular design that hews more closely to the concerns and quandaries of computer-related fields of 
study and work” (p. 59), hence to media studies and education. SF has real-world interest and excitement built in; it 
is ripe for use to engage and excite students about STEM subjects. By relating science to reality and using highly 
popular literature to introduce it, educators strive to make STEM material more relevant and more accessible for all. 
Despite socioeconomic, educational, gender, and racial factors as contributors to the general decline of STEM 
interest and the exclusivity of STEM clubs and organizations, working with SF does have a positive effect on 
student attitudes toward STEM subjects (De Lepe, et al, 2015). 

These are all compelling lessons and ideas using SF from practitioners and academics working with 
practitioners, but there is much less evidence that middle schoolers would benefit in the same ways due to their lack 
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of context of some issues or situations that more mature students may be able to more easily grasp. People are 
motivated to pursue and engage in certain activities based on two primary factors: desire for accuracy and desire for 
a particular conclusion (Kunda, 1990). Middle schoolers will develop interest in reading SF and in STEM activities 
because they think it is the right thing to do and/or because it is the desirable thing to do; the latter is more likely to 
lead to prolonged interest (Renninger & Hidi, 2016). Prolonged interest needs access to feed it, especially among 
underrepresented populations. “A vital question to answer is how to engage underrepresented young people in 
STEM and help them persist in these fields as they progress through their education and professional trajectories” 
(Subramaniam, et al, 2012, p. 162). 

Scientists and science teachers generally like and appreciate SF because it delves into worlds they enjoy 
thinking about and suggests the possible futures and consequences of their actions (Luokkala, 2014; Freedman & 
Little, 1980; Marks, 1978). However, there is a disconnect between scientists and science teachers and home life; 
many underrepresented youth do not have STEM role models at home. Without role models in STEM fields to guide 
them, many middle schoolers have little opportunity to develop the scientific literacy needed to succeed in 
increasingly-difficult classes (Ahn, et al, 2014). The concept that SF can help to demystify science for students is 
almost certainly one of the reasons fans enjoy it so much (Brake & Thornton, 2003). More attention should be paid 
to students before eighth grade because those years may be more formative to their development of scientific 
literacy (Tai, et al, 2006). School librarians can fill that role. 


Role of the School Librarian in STEM Education 

The school librarian should be the captain of the research team, the conduit through which the direction of 
projects flow. Subramaniam, et al (2015) present a case for “. . . the unique contributions that [librarians] can make 
to young people’s learning of science” (p. 3) and other STEM fields. This is the link to librarianship that drives this 
study. Most students can Google an answer, but concerns arise about unprepared STEM researchers in a Project- 
Based Learning (PBL) environment that stresses the use of technology, sourcing, and collaboration among class 
members and potentially others: Who is teaching them to self-reflect, and what is the value of reflection? Who is 
teaching resource and content vetting? (Harada, Kirio & Yamamoto, 2008). Specifically addressing middle-level 
students (ages 8-12), Subramaniam, et al (2015, “Simple”) argue for new digital literacy programs in an 
environment more conducive to “. . . building on tweens’ existing heuristics and thereby resulting in strategies that 
are simultaneously compatible with their natural inclinations within the online environment and likely to 
consistently lead them to accurate credibility-related judgments” (p. 550). Subramaniam, et al’s (2015, “Simple”’) 
study illustrates the digital divide, perpetuated by socioeconomic factors, that still exists. Such gaps as the lack of 
on-grade literacy and reading skills, effective methods to assess online credibility, and lack of interest in accuracy in 
favor of search result matching key phrases stultify the effectiveness of research and learning. This is a time when 
school librarians are needed more than ever to teach and promote digital literacy as effective teachers (Mardis, 
Kimmel & Pasquini, 2018). The problem with school librarians as school leaders and teachers is that few 
administrators understand and appreciate what school librarians can do and what they are capable of bringing to 
schools (Mahoney & Khwaja, 2016), so it is unlikely that they will be assigned leadership roles unless they advocate 
so much that they are unable to perform their daily duties. Although school and district leaders may value media 
literacy, concerns about lack of funding, teacher training, and an uneven spread of technology across the district are 
still major factors in decision making (Mahoney & Khwaja, 2016). Time constraints on teachers and school 
librarians hinder collaboration (Rawson, Anderson, & Hughes-Hassell, 2015). Also, school librarians are not always 
encouraged to teach due to multiple building responsibilities (Zmuda & Harada, 2008). 

The style of connected learning that Subramaniam (2016) describes is ideal for engaging students in STEM 
subjects and related interests, since many of the activities associated with STEM engagement occur outside of the 
classroom in informal environments such as social media platforms, clubs, and museums. It follows the Future of 
Library Services for and with Teens report that advocates for libraries to promote “. . . the three spheres of learning 
(interest-driven, peer-supported, and academically-oriented) among non-dominant teens” (Subramaniam, 2016, p. 
2). Subramaniam suggests using Radical Change theory, a blend of the digital concepts of interactivity, connectivity, 
and access to promote growth and improved response to teens by librarians. By examining changes in librarians’ 
interactions with teens, a new and more responsive method of engagement can be developed. Her method features 
the analysis of 23 cooperative learning projects resulting in three themes, all working with children aged 5 to 17: “.. 
. the foundation for the cooperative inquiry method, a selected cooperative inquiry technique or techniques involving 
children/teens in the design of technology or learning programs, and an extended explanation of the choice of 
cooperative inquiry technique in the design of specific technologies and learning programs (beyond simply saying 
that they used a selected technique)” (Subramaniam, 2016, p. 7). To achieve success, not only do students have to 
adapt to changes in information content needed in their cognitive toolbox, but also to changes in learning styles and 
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formats, depending upon societal forces such as technology advancement, sociocultural shifts, and educational 
trends such as statewide testing. This is a socially constructivist view of learning incorporating a Vygotskian, 
contextual Zone of Proximal Development (the area between what a person believes she knows about a subject and 
what she actually knows, and the space in which mentorship is enabled) that aligns with current theories on the ways 
in which students construct information and seek assistance from mentors and peers. Since students are growing up 
in a networked world, a networked learning philosophy seems logical and cogent. 

Every middle schooler is entitled to the tools she needs to succeed in an uncertain future. School librarians 
need to be a part of that future. They do not merely create an environment in which higher standardized test scores 
become more likely (Lance & Hofschire, 2012). The school librarian is the only individual in the building “. . . 
tasked with building and maintaining a collection of diverse, high quality, current resources that support curriculum, 
complement adopted texts, enable professional learning, and pique student interest” (Mardis, 2014, p. 250). AASL 
defines the five primary roles of school librarians: information specialist, instructional partner, teacher, program 
administrator, and leader (American Association of School Librarians, 2014). No one else in the building is capable 
of this range of capabilities. Additionally, and probably in response to a growing movement, they describe 
librarians’ possibilities of creating semi-formal learning environments in school libraries, excellent choices for 
librarian collaboration since they can offer more media, more guidance, and more interest-driven topics and themes. 

Collaboration is crucial to building success; no educator should be an island (Meyer, 2017). Teachers and 
school librarians, when working on professional or curricular development, characterize the relationship as a 
partnership of equals, with teachers providing subject expertise and intimate knowledge of their students and 
librarians providing information literacy expertise, knowledge of resources, technology expertise, and guidance to 
students through the conceptual and emotional challenges of the research process (Yukawa & Harada, 2009). There 
may also be an overly high degree of confidence that simply connecting to learner interests or creating and 
participating in an active culture of science enhances science learning and connections to STEM fields (Yip, et al, 
2014). The school librarian is the link the binds all elements of the school together, and she has no claims to 
information ownership. Above all of her functions, the school librarian is most importantly the only person in the 
building certified and trained to understand and teach information search processes and practices. Not only do 
school librarians increase reading scores, their absence nullifies that effect in a major Colorado-based study (Lance 
& Hofschire, 2012). This is crucial, since a lack of information search, assessment, and organization can profoundly 
affect adolescents’ self-efficacy and potential for a long and successful life, exemplifying the need for librarians to 
teach those skills (Subramaniam, et al, 2015, “Bit’). 

Defining the role of school librarians in enhancing science learning, Subramaniam, et al (2015, “Role”) 
conclude that school librarians should be playing a greater role in science learning because they can: 

1. encourage young people to engage in authentic inquiry practices by teaching them about information 

search models and strategies; 

2. engage young people’s everyday life interests by linking science learning to media and technology that 

appeals to them or they “see themselves’ in; and 

3. promote the norms of ethical and social interaction in sharing science knowledge. (p. 10) 


Guided Inquiry 

An important companion to the explosion of technology access on school campuses at all levels is guided 
inquiry design. Kuhlthau, Maniotes, and Caspari (2012) describe guided inquiry as both grounded in best practice 
and research and crucial for successful living and transition to the workplace. Guided inquiry is authentic to students 
because they work with peers and with chosen, not assigned, subjects, or at least chosen sub-topics within a 
curricular theme. Technology’s ability to enable users to simultaneously create and share, plays an important role in 
successful critical skill building in students, the ultimate goal of guided inquiry design. Without guidance, students 
often approach the process as a simple collecting and presenting assignment that leads to copying and pasting with 
little real learning. With guidance, students are able to concentrate on constructing new knowledge in the stages of 
the inquiry process to gain personal understanding and transferable skills. Students’ feelings play an important part 
in the constructive process of inquiry that indicates a zone of intervention for teachers and librarians. For example, 
students get frustrated in the exploration stage of inquiry and need encouragement to take time to read and reflect 
and guidance in making sense of information that does not fit together smoothly. Guided Inquiry, based upon 
Kuhlthau’s (1991) information search process (see Figure 1), is intended to be collaborative, engaging, reflective, 
and iterative, as students explore new ways to solve problems, answer questions, and create artifacts. Motivation, 
interest, relevance, and affect are primary factors in engaging in guided inquiry learning. 
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Model of the Information Search Process 


Initiation Selection Exploration Formulation Collection Presentation Assessment 
Feelings Uncertainty Optimism Confusion Clarity Sense of direction/ Satisfaction or Sense of 
( Affective) Frustration Confidence Disappointment — accomplish- 
Doubt ment 
Thoughts vague. ————_ focused Increased 
o : —_—""):... >> = 
(Cognitive) self: 
increased interest awareness 
j seeking _ relevant information seeking ertinent information 
Actions 2 : 2 P ; 
(Physical) Exploring Documenting 


Figure 1. Kuhlthau’s (1991) 6-stage, 3-domain ISP Model 


Guided Inquiry prioritizes and defines that space for students and staff alike by providing opportunities for 
engagement and reflection (Kuhlthau, Caspari, & Maniotes, 2015), especially since students often mis-assess their 
searching skill and then wrestle with the cognitive, affective, and behavioral ups and downs predicted by Kuhlthau 
(St. Jean, et al, 2015). Self-reflection helps students and teachers during this time, and both benefit from activities 
like log-keeping (Harada, 2016). 

In order to map out the information search process using guided design theory, Kuhlthau, Maniotes, and 
Caspari (2012) designed a new 8-step ISP model to augment the previous 6-step ISP cited above. The newer model 
includes opportunities for students to practice the guided inquiry behaviors of analysis and sharing. 


Arrows 
indicate 


collaboration 
opportunities 


Fig. 2. 8-step Guided Inquiry Design Process (from Kuhlthau, Maniotes, and Caspari, 2012, p. 2) 


Guided Inquiry Design Process 


The Exploration stage, here called Explore, is basically the same as it was in the original model. Students 
are encouraged to browse and scan many sources and consider themes and ideas rather than simply gathering facts 
and figures. It is possible that Google Answer Boxes are serving to affect this phase of the guided learning process 
insofar as students may not be experiencing the confusion and oversaturation of information indicated by Kuhlthau, 
Maniotes, and Caspari (2012) because their trust of the Answer Box supercedes their tendency to overwhelm 
themselves with too much information from too many sources (Miklosik & Dano, 2016). 

Collaborating with a Science teacher for a semester in a middle school Robotics class has demonstrated to 
me that researching with students ina STEM environment is an ideal location to explore these questions. SF film 
and television clips, as well as short stories, poems, and potentially novels (if time allows) and fan fiction (age- 
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appropriate) related to one or more of Luokkala’s seven foci (2014) are ideally suited to motivate students to 
become more engaged with STEM-relate activities and interests, and curricula featuring SF as an entry point into 
STEM interest and blending discussion, artifact creation, reflection, and familiar contexts should be considered and 
implemented to maximize students’ STEM potential. Interventions should be developed by practitioners until there 
are enough SF-based curricula available to all students. 


Conclusion 

Collaborating with a Science teacher for a semester in a middle school Robotics class has demonstrated that 
the power and influence of SF research with students ina STEM environment would be an ideal location to explore 
these questions, and such research should be conducted with interviews, surveys before and after the experience, 
language arts interventions, and creation of artifacts related to the material. SF film and television clips, as well as 
short stories, poems, and potentially novels (if time allows) related to one or more of Luokkala’s seven foci (2014) 
are ideally suited to motivate students to become more engaged with STEM-relate activities and interests, and 
curricula featuring SF as an entry point into STEM interest and blending discussion, artifact creation, reflection, and 
familiar contexts should be considered and implemented to maximize students’ STEM potential. Interventions 
should be developed as the project progresses. Special attention will be given to collecting pre-and post-intervention 
data to demonstrate the potential effectiveness of the use of SF ina STEM environment. 

Two areas that have been touched upon in this study should be investigated and considered further. Use of 
Guided Inquiry during STEM and SF work is one area that deserves greater scrutiny, as several researchers cite the 
social constructivism and intergeneralizability involved with researching and creating around specific topics or in 
specific communities. Also, access limitations to STEM-related activities and fields due to culture, race, gender, and 
ability level need to be addressed to ensure equal and accessible education for all students, and equal access to 
potential future success and the maximization of the natural abilities and talents that all students inherently possess. 
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Abstract 


Realistic hand drawing is still a valuable skill for the designer, and learning in an in-person drawing course 
was enhanced through the use of Augmented reality. Representational drawings of interior spaces were digitized and 
displayed in the building environment using AR tags. This let the students envision how a space course be 
represented in the same environment they were drawing. AR-based graphic instructions for the construction of 
perspective drawings were also developed and shared with students. Examples of student drawings are included in 
this writing along with placement in the represented spaces. The research question was whether the use of 
augmented reality could increase the drawing ability of beginning design students in creating two-dimensional 
drawings from three-dimensional space. 


Introduction 


Two of the latest technologies to enter the educational realm are Augmented Reality (AR) and Virtual Reality 
(VR). Both have received considerable media attention over the past few years. Differentiating the two is the level of 
immersion as well as the software and hardware needed to create various experiences. 

Augmented reality was used for implementation as there were pragmatic advantages over virtual reality: The 
only hardware needed as a smart device, and most college students have smart phones (Roberts, Yaya, & Manolis, 
2014). In 2018 the Pew Research Center found 94% of adults aged 18-29 owned a smartphone (Pew Research 
Center, 2018). Virtual Reality however, is supported by more expensive hardware including the HTC VIve or Oculus 
Rift technologies. Even the Google developed affordable headset made of cardboard needs additional equipment. 

Augmented reality has made inroads into education through a number of applications. AR has been used to 
train surgeons and teachers have used AR to explore different environments within the classroom. 

One use of augmented reality was to represent costly mechanical parts with projections of bearings and gears 
in different views of these objects. This reduced “the time, effort and monetary cost needed for developing various 
educational materials for teaching hand-drawn mechanical drawing” (Horii, & Miyajima, 2013, p. 255). This also 
helped develop students' manual drawing ability and a three-dimensional sense of the drawn objects. 

Augmented reality has been used to project objects into the physical space rather than viewed through the 
screen of smart device. This was a use of Spatial Augmented Reality, or SAR, an enhanced version of AR, which 
sought “to improve speed or ease [of] drawing by projecting photos, virtual construction lines and interactive 3D 
scenes” (Laviole, & Hachet, 2012, p. 175). 

Use of augmented reality in this manner adds additional richness to education and. Helps to accommodate the 
different learning styles of students. This writing concentrates on a basic application of augmented reality in a 
beginning drawing course for students in apparel, graphic, or interior design. 


Augmented reality and beginning drawing 


A studio class was supplemented by the use of augmented reality. The course is a required drawing course 
for undergraduate students in interior, apparel and graphic design at the University of Minnesota. The focus of the 
course is the development of drawing skills for design thinking and representational drawing. Augmented reality has 
benefited drawing instruction in our courses. The course must serve as the basis for all graphic design students as it is 
the only drawing course required over the four-year program. Course enrollments average 20 students in each section 
of the course. 

Initial drawing skills of students vary greatly in every section. Surprisingly, a large number of entering 
design students have little or no drawing experience or skill. For those who have difficulty with drawing, augmented 


367 


reality can be a way to improve their skills in short period of time. Similarly, it can also enhance the abilities of more 
skilled students. AR levels the playing field as the technology by focusing on the resulting idea as opposed to the 
manual skills necessary to execute the drawing. 

One major project that has been a centerpiece of the course is a drawing of an interior space in the classroom 
building. This drawing is also a requirement for a subsequent mandatory portfolio review for all design students. The 
final drawing, rendered by hand in pencil on paper is a full value representation, with values ranging from black to 
white. It shows a significant expanse of three-dimensional space. 


10:40 AM 


Figures. I & 2. Images located in building environment 


The linear perspective drawings are created by students while sitting around the building in various spaces. 
Photographs are sometimes used to record the lighting in the spaces for a given time, but the development of drawing 
skills in situ is the primary goal. 

In previous semesters, good examples of student work has been shown on a projection screen in the 
classroom. Augmented reality, however, allows the drawings to be seen in the environment where they were created. 
This lets students see how a space has been illustrated in the past and from the same viewpoint. 

Zapworks is a vendor that offers an online service for augmented reality that is accessed through their mobile 
phone app. This was used to overlay digital images onto the actual space. Drawings by previous students could be 
seen in the actual spaces that are represented. It is an exhibition of pencil drawings allowing students to view the work 
of their peers, and how they represented spaces by using AR. See Figures | & 2. 
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mcneal_hall_0007. 
Sit in the middle of the atrium by 
the column closest to the main build- 


ing entrance on the ground floor 
looking toward the wide staircase to 
the first floor. 


Fig. 2. Typical identification code [NOT ACTIVE] 


The work could be viewed by students by downloading the free app for their type of phone. The 
identification symbols that accessed the digital images were posted in different parts of the building. 


Drawing construction 


Developing correct representation of perspective is an important early skill of learning to draw. Future 
development of augmented reality includes visual identification of structural components necessary for one-, two-, 
or three-point perspective rendering. This includes elements such as horizon lines, vanishing points, converging 
lines, station points, ground line, ground plane, picture plane, and center line of vision (Gill, 1974). Sighting 
techniques and angle gauges were also identified as possible future enhancements. These elements would allow 
students to see how drawings could be constructed by hand to create a realistic representation of space. 


Exhibition 


Students were able to "mount" an exhibition of their work while maintaining the safety of their original 
drawings; there was no need to print large representation quality copies of their work. Student work was illustrated 
and tagged throughout the spaces of the building for the public to view through augmented reality. 

Results of the course will be compared with previous versions in terms of quality of work, student 
acceptance, and impacts on the learning process. Additional ways of applying augmented reality to the design 
curriculum will be explored. 
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Figure 4. Placard with AR code 
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Descriptors: Instructional Design, Graduate Studies 
Introduction 


Instructional design is an interdisciplinary field where many professionals come from a variety of academic 
and professional backgrounds. Instructional designers often are instrumental in workplace learning and higher 
education. While the job title and duties may vary from organization to organization, typically instructional 
designers create learning experiences for students, faculty, facilitators/trainers, customers, and employees based on 
organizational and learners needs. These learning experiences can be created for face-to-face, blended, or online 
learning formats. Instructional designers are often expected to be experts in curriculum development, learning 
theories, and learning technologies. 

Formal instructional design programs have been around in the United States since the 1950s ranging from 
various program names including Information Technology and Curriculum Management (Gustafson, 2001). Despite 
these programs being around for more than 50 years, there is a lack of formal professional standards across the 
profession. In other words, a graduate from one instructional design program may be better versed in learning 
theories where another graduate from another program may be better versed in assessment. Furthermore, many 
instructional designers fell into the profession after being promoted from subject matter experts to instructional 
designer. Because of this, many accomplished instructional designers do not hold formal degrees in this field This is 
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one way that instructional designers differ from educators, who have legal provisions for licensure and education. 
This range of skills can be seen in various job descriptions for instructional designers. As a result, there are no 
uniformity in knowledge and skills required from position to position. 

Due to the diversity of backgrounds of instructional designers, it can be challenging for these professionals 
to have a baseline knowledge of instructional design (Brown & Green, 2015). Furthermore, the golden standard of 
instructional design and what is and is not acceptable often varies depending on the organization that employs the 
instructional designer. One study found that over 70% of job announcement for instructional design positions 
require a bachelor’s degree and 50% require | to 5 years of experience (Kang et al., 2015). An analysis by this 
paper’s co-author found that 56% of the job postings analyzed required applicants to have a competency in specific 
software packages (North, 2019). Instructional designers must be comfortable working with people, designing 
instruction, developing or using tools for learning, and pulling and interpreting data for decision making. 

Working in a profession that has been revolutionized by technology by delivering eLearning and virtual 
instruction, it is often difficult for instructional designers to keep up with the rapid pace of technological 
advancements (Reiser, 2017). Furthermore, with many instructional designers not having a formal professional 
development budget, often they are on their own to develop new skills and upskilling themselves (Cheong, 
Wettasighhe, & Murphy, 2006). To develop these skills, some instructional designers pursue higher education 
programs. According to West et al., institutions that offer programs to teach instructional designers are typically at 
the master’s and Doctoral level (2017). Therefore, instructional designers seek programs that will help them to grow 
professionally. The programs available by institutions vary. 

Programs designed to educate instructional designers should provide them with a basic knowledge of 
applicable theories and a portfolio of tools to release on the world. However, we know that places of business, 
including academia and industry, have explicit needs that they expect each prospective employee to have beyond 
theory. Industry expect their employees to have a working knowledge of current tools and be able to hit the ground 
running. 


Purpose 


The purpose of the panel discussion was to review the current needs of the instruction design realm from 
the perspective of graduate students, academic program leaders, and prospective industry employers. The remainder 
of the paper is a summary of the panel discussion and the insights from the panelists from the Association for 
Educational Communication and Technologies 2019 Inspired Conference held in Las Vegas, NV on October 24, 
2019. 

Participant Discussion 


Question: Please introduce yourself and share you background with the audience. 
Cara North: My name is Cara North and I’m a Senior Learning Experience Design at The Ohio State University. I 


am also a Ph.D. student in Educational Studies with an emphasis in Learning Technologies at The Ohio State 
University. I hold a BA in broadcast journalism from the University of Kentucky and an MA in workforce 
development from The Ohio State University. 


Anna Leach: I am a Teaching Associate and 2nd year PhD student at the School of Information at the University of 
Arizona. I have a bachelor's degree in Mathematics from Seton Hill University and a Master of Learning 
Technologies degree from the Ohio State University. 


Tracy Shroyer: I am a Learning Consultant in the financial services industry. Although I have been in the corporate 
world for over 23 years now, my experience working in a Learning & Development role has only spanned the last 
three years. I have also had the opportunity to hire and work closely with instructional designers as contractors for 
my team. I have a Masters in Business Administration, a Masters in Industrial-Organizational Psychology, and a 
PhD in Organization and Management, specializing in Leadership. 


Ana-Paula Correia: I am an associate professor in learning technologies in the College of Education and Human 
Ecology at The Ohio State University. I am also the director of OSU’s Center on Education and Training for 
Employment. 


My research interests are related to distance education, online and mobile learning, collaborative learning and 
different approaches to instructional design education. At OSU I’ve founded the Learning and Experience Design 
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Research Group, a group dedicated to investigating learning design processes and pedagogies to create deep 
learning. I am an active contributor to the AECT, serving as President (2011-2013) for the Research and Theory 
Division and member of the editorial board for Educational Technology Research and Development (2015-2018). 


Question: Describe your professional experience in instructional design. 


Cara North: I’ve been working in the broad scope of learning and development for all of my professional career. 
After I graduated from the University of Kentucky, I moved back in with my parents because I couldn’t find a job. I 
saw an ad in the local newspaper for guaranteed interviews at a telemarketing center in Huntington, WV. I went to 
the interview and walked out with a job. After working at the center for a few months, I was promoted to a call 
center quality assurance job. In this job, a small part of the duties was training call center employees about the call 
center policies and procedures. After my first training class, I knew I wanted to work in this capacity for my career. 
Throughout my career I have worked as an instructor led trainer, a curriculum developer, an LMS administrator, a 
training coordinator, and an instructional designer. 

I’ve worked as an instructional designer at Amazon.com as well as The Ohio State University. While at 
Amazon.com, I made training materials for call and chat customer service representatives all around the world. My 
materials included standalone eLearning modules in our LMS, training guides and curriculum for customer service 
onboarding, and performance checklists for customer service managers rating their associates on various technical 
and soft skills. In my current role as a senior learning experience designer at The Ohio State University, I manage 
the LMS and courses for the College of Education and Human Ecology. Much more so than my corporate 
instructional design job, a good portion of my job is providing learning technology support to faculty, including how 
to use conference call platforms, apps on iPads, and general LMS how-to support. My skills and involvement with 
courses are optional, meaning the faculty member has to come to me for me to support their course. 


Anna Leach: I too fell into training and instructional design quite by accident. While working as a data analyst at 
the Ohio State University, we went through a system change. The change required new database instances and 
understandings which meant training on the new system and new ways of reporting. Training on the new system 
and the database that housed the system was designed and facilitated by contractors. Eventually the contractors left 
and a few of us were left with picking up the training. We had no formal training experience. Our trainings were 
brief introductions to the data and opportunities to explore the data using different analytical tools. In some 
instances, it was easy but in many others it was difficult. Many people were frustrated with the tools or had trouble 
with the data. Fortunately, one of my colleagues was a former teacher. She exposed us to ideas and ways of 
instructing. From here, I explored instructional design. 

I have always had an interest in continuing my education, so when the Master of Learning Technology program 
started at Ohio State University, I applied. After completing the program, I decided to continue on in my studies to 
a PhD program at the University of Arizona. I enjoy the research and study of educational technology. My focus of 
study is in the data produced by educational technology and how it is interpreted. 


Tracy Shroyer: I have always wanted to get into a training-related role and have had opportunities here and there in 
roles within the organization I work over the years. In the fall of 2016, I was able to secure a Learning Consultant 
role within my organization and was over the moon with excitement. Since being in this role, I have had the 
opportunity to develop the learning strategy for a group of 260+ employees in my area of the business, to develop 
learning programs and curriculum. In doing this and working with a group of instructional designer contractors, I 
found a need at one point to jump in and learn how to use eLearning design software and develop about a dozen 
modules on my own to ensure our group achieved results. Recently, I introduced the concept of microlearning, and 
we are working on changing the method of learning in two content areas to leverage microlearning. We are now 
looking at the appropriate medium to deliver this learning. 


Ana-Paula Correia: I have experience in K-12, academia, and industry as an instructional designer. Being a K-12 
science teacher for several years before pursuing my graduate studies gave me a practical perspective of 
instructional design as a field of study. I have also worked as an instructional designer for a large pharmaceutical 
company for a couple of years. This experience allowed me to translate into practice what I have learned as a 
doctoral student in Instructional Systems Technology at Indiana University. I have been working as a college 
professor since 2005. My professional experience as both practitioner and academicist has provided me with insights 
that I apply every day on my practice. 
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Question: What knowledge or skills are employers looking for in instructional design graduates? 


Cara North: I think it depends on the setting the student wants to work in. Having worked in both corporate and 
higher education settings, I will provide guidance for both. 

Higher Education setting 

Instructional design in higher education needs a disruption in my opinion. It should not be optional for courses to 
use an instructional designer, it should be mandatory. I’ve seen many travesties of online courses out there than 
make the student experience poor and frankly reflects poorly on the university at large. Given the current culture of 
instructional design in higher education I would say an instructional designer needs a significant depth of knowledge 
about learning technology tools. These tools include using conference call software, apps, and various programs and 
knowing how they look/function on multiple devices such as Windows, Mac OS, tablets, and mobile. While many 
may assume that information technology help desk may help with this, these questions in my experience come up 
way too often to higher education instructional designers. Furthermore, project management skills are critical to the 
success of a higher education instructional designer. I cannot tell you how many times I’ve been “ghosted” by a 
faculty member who wants help with a course and then disappears from email. Being able to determine if a project 
should keep going or should be taken off the load for the semester is critical. Furthermore, higher education 
instructional designers should have a good grasp of learning theories and research. Often faculty will challenge you 
on your design decisions and being able to back up perspectives with cognitive load research, behaviorist or 
constructivist perspectives, and even explain your design process is critical to getting their buy-in. 

Corporate setting 

A corporate instructional designer should be able to design a learning experience that is performance based. By 
creating a learning experience that is performance based, this can tell the organization exactly what the person 
taking the training will be able to do and apply at the end of the experience. Furthermore, knowing how to use 
various tools such as eLearning authoring tools and video editing is critical. There are two big eLearning authoring 
tools primarily used in the US market: Adobe Captivate and Articulate Storyline. I think it’s necessary to be able to 
build a module in each of these tools as it is likely you will have access to one or the other at your job. 

Knowledge and skills for both settings 

If you are going to be in the profession of instructional design, it is critical that you are a lifelong learner. 
Throughout the eleven years of my career, I’ve seen so many changes including the way eLearning is created to 
alternative ways to collect user data. This is exciting and I try my best to keep up with what is going on in the 
profession. Additionally, the ability to keep the user in the middle of the design is critical. Your learning 
experience should be user-centric, and you won’t know what they want if you don’t ask them. 


Anna Leach: Based on the literature that I have been exposed to, instructional designers are meant to wear many 
hats. They need to be able to collaborate and work well with people. They must be able to communicate well. They 
need to be able to work with many software packages and be able to keep up with or learn new tools regularly. They 
must also have a foundational knowledge in learning theory. But I would urge people to also start being 
comfortable working with data and focusing on data literacy. I do not believe that all instructional designers will 
have to create reports or pull numbers, but I think it is an asset to be able to demonstrate, numerically, their work. 
For example, in sales, a salesperson can show the number of sales from month to month to show success. In 
instructional design, what metrics do you have to be able to show your worth or show that the instruction that you 
designed is working. Simply collecting feedback from your users provides a data point that can be used and shared 
with others. Instructional designers almost always begin with a needs analysis before creating the training. Here 
you will have objectives. Follow up with the training implementation to see if those objectives are met. This is 
another data point that can demonstrate the value of the work. 


Tracy Shroyer: I read a great article recently that talked about 10 skills that will allow learning and development 
professionals to get ahead in the industry (Spinelli, 2019). These skills include project management, change 
management, leadership, management development, data driven decision making and analysis, communication, 
strategic and critical thinking, marketing skills, relationship building, and teamwork and collaboration skills. In my 
experience hiring and working with instructional designers, having more experience in several of these skill areas 
would absolutely allow the individual to bring more value-add to the organization in which they are working or 
contracting. Instructional designers, as I would say with anyone in a focused role, need to take a step back and look 
at the bigger picture and skills needed in the business world to be more effective in being successful and achieving 
results. 
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Ana-Paula Correia: Employers want to see what new professionals can do with their academic degrees and 
certifications. Instructional design graduates need to be able to show what they can do with that knowledge and 
skills by creating a digital portfolio that showcases their skill set. One aspect of the master of Learning Technologies 
that our team at The Ohio State University is particularly proud of is the Practicum experience. In every course, 
students create learning artifacts that are included in a portfolio that is developed as part of the Practicum. However, 
the biggest strength of the Practicum experience is to work with a real-world client, identify an instructional problem 
and address that problem with a tangible solution. The Practicum is a structured online experience carefully guided 
and supervised by the student’s advisor. It includes five distinctive milestones, as, establishing a memorandum of 
understanding and regular progress reports to the advisor and client. 


Question: What recommendations or solutions would you like to share to help improve instructional design 
education? 


Cara North: I feel there needs to be more practitioners involved in instructional design education. Not everyone 
who is seeking an instructional design degree wants to be a researcher. Researchers will not be able to help 
instructional design students often build interactions. In order to keep up with the rapid pace of change, 
instructional design programs should partner with practitioners or local Association for Talent and Development 
chapters to help build the technical acumen of the instructional design students. 


Anna Leach: Project-based learning. Instructional Design programs need to offer opportunities to implement and 
apply knowledge. These opportunities held to build portfolios. Portfolios are critical for demonstrating work. Also, 
exposure to instructional design tools. A student should have a safe-space to work on a project in the current and 
known tools of the trade; things like Adobe Captivate, Articulate Storyline, etc. I would also like more multimodal 
projects; working with video, audio, and the like. If the program has requirements that the graduate leaves with a 
foundational knowledge of learning theory as well as practical experience managing projects and working with 
relevant tools, the program will produce a valuable and skilled employee. 


Tracy Shroyer: I absolutely agree with what Cara notes about integrating more practitioners into instructional 
design education. This needs to occur more in every field, in my opinion. Practitioners provide the reality of putting 
the theory into practice and day-in and day-out successes and challenges of the role. As I mentioned before about the 
skills learning professionals should find ways in which to grow, I highly recommend the Association for Talent and 
Development (ATD) chapters as a place in which to learn and grow. Being part of the national and local chapter of 
ATD this past year has provided me with the opportunity to enhance my technical knowledge of learning tools, and 
to expand my professional network. I have also learned the importance of having a portfolio through my work in 
hiring instructional designers. Some people interview well, and this enables my company to get real-world examples 
of what they have done and learn more about how they went about the process of designing something as well. A 
portfolio also provides instructional designers with the opportunity to showcase their skills beyond the technical 
design aspect, such as project management and collaboration. 


Ana-Paula Correia: Instructional design education, and education in general needs to evolve into rich and authentic 
learning experiences. For example, internships, employment during college, volunteering, and extra-curricular 
activities provide opportunities for people to showcase how they apply their knowledge and skills, make decisions, 
and create solutions in real working situations. 


The purpose of these learning experiences is for students to have opportunities to engage in real-work experiences in 
a variety of formal and informal settings and strengthen their professional portfolio. They may consist of: (1) 
participating in the development of an educational product through various activities (e.g., conducting a needs 
assessment, reviewing various prototypes, assisting with the design); (2) assisting with design and development of 
training modules; or (3) assisting leadership on making learning technology-related decisions. 


Question: What is the future of instructional design education? 


Cara North: I would like to see instructional design to an apprenticeship model. In order to become an 
instructional designer, you should go through a project start to finish. There are so many elements of a project to 
manage and things that can’t necessarily be taught including how not to break the heart of a subject matter expert 
who insists all of their content needs to be translated into eLearning. 
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Anna Leach: | worry that the future will be the same as it is now because the programs aren’t changing rapidly 
enough. My experience from looking at the literature is that the programs focus on theory and not enough practical 
application. I am hopeful that institutions will start working within and outside of their own institutions to create 
projects for students to work on that are meaningful. 


I also think that a divide in instructional design is coming. A divide from those that work with the technology and 
those that design. I worry about this direction. From experience in corporate settings, when functional and technical 
are not in communication, gaps are created. Ifa divide in the field starts, like it has with titles like learning engineer 
versus instructional technologist versus instructional designer, what will happen to the instruction? Will one half 
focus on tools use and implementation while the other focuses on learning theory? 


Tracy Shroyer: I am unsure of what the future of instructional design education looks like, but I know there is a 
desperate need to do more than just feed design theory and the technical skills to students. Students should graduate 
from an instructional design program being more well-rounded to enter into the field and take off running and 
adding significant value to organizations. 


Ana-Paula Correia: One approach for instructional design education that was proposed by one of my former 
doctoral students, Farrah D. Yusop, and I is called the Civic-Minded Instructional Designer (CMID) framework 
(Yusop & Correia (2012). This framework asserts a paradigm that any instructional work has social implications. 


As a result, an instructional designer is viewed as a potential agent of social change who has the transformative 
power to bring about good to the society at large. The term “civic-minded instructional designer” is proposed, 
referring to an instructional design professional who: (a) has the public interest and a sense of civic responsibility at 
the forefront of his/her work, (b) is attentive, responsible, and responsive to the emergent instructional needs of the 
members of the community, and (c) utilizes his/her knowledge and skills in instructional design and technology to 
improve learning and performances of others. This framework may be an avenue to inform the future of 
instructional design education. 


Question: What is one piece of advice you would give someone entering into the Instructional Design field? 


Cara North: Network. Network. Network. You need a community of people to help lift you up and keep you on 
target for your professional development 


Anna Leach: Data literacy. Explore ways of collecting and analyzing data from your ed tech tools. 


Tracy Shroyer: Continually work to build and expand your skill set (beyond the technical design realm) and build 
your professional network. 


Ana-Paula Correia: Be knowledgeable. Be able to explain the rationale for your design decisions and the strategies 
you use to overcome the design constraints. Use a rationale rooted both in theory and practice. 
Keep studying. Stay curious. 


Summary 


The panel discussion highlights the breadth and depth of employment in the instructional design field. 
Recent graduates of instructional design programs find themselves ready for entry-level positions, but our panelists 
encourage programs to help future enrollees. They share that those in the instructional design programs will need to 
prove they can do project management, collaborate, communicate well, use relevant software, and more. There is a 
high demand for instructional designers to be eager life-long learners. 


Future Studies 
The participants would like to further explore the current needs of instructional design with respect to what 
employers are looking for as well as what institutions, in the United States and internationally, offer in their 


instructional design programs. We are interested in understanding the many titles of instructional design job 
postings. We are interested in studying job posting requirements. We would like to converse with others in 
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instructional design at different institutions of higher education as well as corporate environments. We also want to 
study the difference in front line worker training instructional designers versus corporate mandated training 
instructional designers. 
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Descriptors: Instructional Technologies, Educational Technology Evaluation 
Introduction 


In formal instructional design, practitioners are trained to complete an evaluation of a technology before it 
is implemented, another evaluation while it is in use, and finally, have methods in place for continuing evaluation 
indefinitely. In higher education, the literature shows different evaluative frameworks for educational technology. 
But, the process of evaluating technology is complex and requires many people to participate and make decisions 
(Kennedy, 2003). Quality frameworks exists for evaluating online courses (Baldwin et al., 2017). However, 
instructional technology implementation in face-to-face or hybrid courses are still being researched and best practice 
have not been formalized. Furthermore, the advancement of technology and constant creation of new tools means 
that the evaluation must be flexible. 

The affordance of these many technologies also means that decisions on tool use in the classroom are not 
made solely at the institution level. From our experiences and the accounts of our colleagues, faculty often find 
themselves increasingly occupying decision-making roles in the selection of instructional technology for the classes 
that they teach or facilitate. Given the nature of course creation, we may not consider formal evaluation of a tool 
before implementation. However, faculty may be “unfamiliar with education evaluation theory and practice” 
(Kennedy, 2003) and have “limited resource variables of time, skill, and support” (Stavredes, 2001). 

Faculty are introduced to these new technologies by their department, colleagues, emails, and conferences. 
Our experience notes an increase of instructional technology in which students must then pay out-of-pocket fees, 
data, or both in order to use the tool and the instructor that decides to use it nor the university do not incur a fee, but 
sometimes even receive a benefit; giving more merit to performing regular evaluations of the tool to ensure it is 
meeting the objectives of the course and the needs of the student. 

The paper will share the experiences of a faculty member, a graduate associate, and a collaborator in 
instructional technologies of implementing and evaluating the instructional technology used, Top Hat. The informal 
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evaluation includes the authors’ perspectives, an interview with the CEO of Top Hat, and the voices of the students 
that used the tool for class. Here we take an exploratory look at the use of student voices to evaluated a recently 
implemented face-to-face and hybrid educational tool, Top Hat. We seek to offer a simple strategy for individual 
faculty and perhaps department level evaluators. 

One component of formative educational tool evaluation is a pilot to ask the consumers of the tool for 
feedback. The students are the consumers of the educational technology that is used in the classroom. Student 
opinions and experiences about the tool should be included in the analysis prior to implementation, the evaluation of 
the effectiveness of the tool during use, and in the follow-up audit evaluations of the tool. We propose using 
mechanisms to gather student voices about the use of the tool in the classroom as a way to evaluate the tool and hear 
from your students. 

Our Story 

After being hired in 2016 to teach large classes at a university, a faculty member and contributing author to 
this paper, wanted to engage the students in her large face-to-face and online courses in ways not afforded by the 
learning management system and clicker technology the university subsidized. A potential solution came in an 
email to the instructor from Top Hat, offering a “comprehensive teaching platform” that enabled hosting and 
presentation of her course content online and face-to-face, along with the ability to question students quantitatively 
or qualitatively. The faculty member, a qualitative researcher, was intrigued. She scheduled a call with a Top Hat 
representative and was surprised to find they also offered her a brand-new iPad if she would adopt the technology 
for a course with 300 students. The iPad was ostensibly for operating Top Hat while walking around the classroom, 
although this was also possible with several other internet-enabled devices she already owned. Although she was 
concerned about the cost of $26 per semester charged to each student for Top Hat, she found the affordances of the 
technology outweighed this cost. She adopted the platform and the iPad. 

Top Hat became a staple of this instructor’s courses for two years, although she was aware that the platform 
sometimes frustrated students. She found the platform’s instructor interface easy to organize and reliable, however, 
students complained that their interface was confusing. She felt confident that Top Hat was an asset to their 
classroom overall, but when students complained Top Hat would not load or overcharged them, she could only 
forward to Top Hat’s support center. Some serious problems proved to be persistent. In her third year of using the 
platform, she noticed students squinting at the screen during face-to-face classes. She discovered that while 
PowerPoint slides were visible on student devices in class, Top Hat’s native slides, called pages, were only visible 
on the classroom screen, severely challenging students with imperfect vision or in a seat far from the screen. She 
wondered how could such a serious problem in the core functionality of this platform have persisted for so long. 

When the instructor received multiple free invitations from Top Hat to their second annual conference in 
2018, she and a team of experts in instructional design decided to research Top Hat more closely and to share their 
findings socially; this is the team of researchers presenting the current work. We decided that the two of us with 
expertise in audio interviewing would attend the conference, record (with permission), and broadcast what we 
learned about Top Hat to students and to other educators through the podcast of our university’s office of digital 
learning. The conference featured a lineup of devoted Top Hat users, proponents of active learning, and venture fund 
investors tracking the latest gains in the adoption of the technology. Conspicuously, none of the students for whom 
the technology is ostensibly developed had been invited to the conference. At the conference we critically 
interviewed educators using Top Hat along with the founder and CEO of the company. 

Inspired by the audio stories collected at the conference, upon returning home, we immediately sought and 
collected a sample of the student perspectives, via an invitation to phone in their evaluations of Top Hat for extra 
credit. The instructors offered students a small amount of extra credit if they called into her voicemail and answered 
a prompt. There were a total of 28 prompts. They ranged in topic from what is your identity on social media and 
does it differ by platform to were you ever bullied on social media. The prompt for Top Hat specifically was as 
follows: Tell a story about a digital “Platforms and You”...What has it been like using Top Hat to participate in 
class(es)? Give us details, stories, frustrations, joys, whatever you've got. 

Out of the 300 students enrolled in the class, 14 students replied. We discovered that while many students 
felt frustrated by Top Hat - including several whose experiences revealed that the company had systematically 
overcharged students - the majority of student evaluations were in support of continuing to use the technology while 
seeking specified improvements, which is the path the instructor selected to follow. Many students appreciated the 
technology and the convenience of actively engaging with the material through the Top Hat mobile app. "I think it 
keeps me involved during class, and I like the overall layout of the website. I like how our textbook is broken up 
into chunks, so it's not just like one book that we have to go through each time," said one student in her testimonial. 
For example, for the word response questions, Top Hat could not display more than three sentences of text. 

Students also complained that the slides were not visible on their computer or mobile device during the lecture. 
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They felt it would have been helpful to be able to see the information on their screen of choice. However, students 
also questioned the additional fee for the Top Hat platform and in the classroom. "We 
didn't know that we had to subscribe to this until we got to class, and we couldn't use our 
bursars' account, which is what I usually put my books on ... it's kind of pricey being 80 
dollars for just one class," said another student, who was also required to pay for a 
textbook through Top Hat. 


provide an inclusive and unique perspective on the adoption of the Top Hat learning 
platform. We produced the podcast episode, which ends with a selection of these student 
perspectives, in December 2018 in the podcast Futures of Digital Learning, available at 
https://odl.arizona.edu/news/2018/12/futures-digital-learning-podcast-engage-2018. 


Considerations for Implementation 


In this exploratory project we asked the students to voluntarily provide us with their perspectives on the use 
of the Top Hat tool in our class. The class is a general elective that surveys the use and impact of social media and a 
majority of the students are freshmen in the University of Arizona eSociety major within the School of Information. 
While requesting, collecting, and disseminating the audio stories, we found several items to consider for the next 
time we ask students for audio feedback. 

Consider your prompt. The instructor of the course carefully considered the instructions given to the 
students and provided a range of prompts. She made it clear that this was for extra credit and not mandatory. She 
also made a clear rubric as to what would be worth full credit and what would not. There was also careful thought 
given to the encouragement of descriptive and specific information without harming themselves or others. Students 
were encouraged to use fake names in the stories. The prompt encouraged the story to be reflective and include 
topics from class material. 

Consider when you ask for feedback; the reflections at the beginning of the semester will be different than 
other times during the semester. If the prompt is asking for experience with an educational technology, the depth of 
experience will vary for each student. Depending on your class structure, offering extra credit through audio stories 
could be used for gathering feedback about a unit or module and the right timing would be during the final 
assessment. 

Consider how you will gather student feedback. A voicemail box has a space limitation. Asking students 
to send a voice memo to an email account might be a better option. Obviously, this is dependent on the size of your 
class. In the instructions for the audio extra credit, we asked students to keep their stories to at least 45 seconds. 
The minimum time allotment probes the student to have a longer and constructed story. Although it was not an 
issue in our situation, most voicemails have a time limitation. Therefore, setting a max time allotment or at least 
warning students of these limitations would aid in the assignment. 

Consider your time. In order to make meaning from these audio stories, someone will have to listen and 
evaluate them; a time consuming process. There are methods for transcribing audio that can aid you in the 
evaluation, but we find that listening to the stories helps us to better know and understand the student. In our 
experience with a large classroom, it is difficult to get to know your students. With audio stories we can hear the 
students voices and it feels like we are getting to know them a little better. 


Conclusion 


Faculty have an opportunity to survey and implement many different educational tools. They are exposed to these 
tools by other faculty, conferences, or their institution which may mean the tool has been evaluated. They may also 
be exposed to these tools via email from the companies that create them. In the later instances, it is less likely that 
the instructor has completed a formal evaluation of the tool. At the time of their selection, the tools are used for 
certain purposes; maybe to engage the students, disseminate information, or collaboration. Evaluation of a tool may 
include quantitative measures like the number of times a tool is used, length of time, log-in times, etc., ; qualitative 
data, like student voices, are useful too. “Quantitative indicators can provide a general overview of a system but 
more qualitative data are needed to interpret what is happening in specific, real-life situations.” (Baron & Bruillard, 
2003). We discussed in this paper an opportunity for simple evaluation using qualitative method of student voices. 
We shared our story and process. We shared some considerations for including student voices as part of the tool 
evaluative process. This process could be used as part of an overall evaluation and is a simple, effective way to 
gather student perspectives. 
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Acknowledgements of Limitations and Areas for Further Research 


We acknowledge that the literature reviewed was not extensive. We are aware of frameworks that indirectly ask 
for student feedback. One framework asks the person that is considering using the tool to consider a context, pilot, 
and report process. The process emphasizes gathering feedback through a pilot study of those that will use the tool 
before implementing (North et. al., 2017); this would be the students. Further, more in-depth study of higher 
education evaluation of instructional technology specifically in face-to-face or hybrid classes is needed. We would 
encourage further study to include the voice of the student as part of the evaluation process. 

Critical evaluations of instructional technology must circulate clearly and often in the same dispersed 
information environments where educators seek and are courted by technology companies. Our experiences and 
collected perspectives from students and others around the platform Top Hat form a case in which inclusive 
evaluation of the medium came later than it should have, but ultimately served our educational communities and 
networked publics as both critical digital literacy and consumer advocacy. The idea of gathering student feedback is 
our responsive contribution toward a culture of critical, education- and student-centered educational technology 
evaluation. We acknowledge the situated nature of our experiences, the limits of reach for including auditory student 
feedback, and the challenges that come with disseminating critical information in environments that encourage 
breadth over depth of attention. We recognize that what we are proposing is but a small component of evaluating an 
educational technology. We intend and invite further research into student pay and other emerging models of 
educational technology adoption, and advocate continued respect toward students’ agency around lived experiences 
with technologies in education. 
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Abstract 


Oral reading fluency (ORF) is important for reading since it can help learners make connections between 
ideas in a passage. However, the process of administering and scoring students’ ORF tests is time-consuming both to 
administer the assessment and to record student performance data. Moreover, computers have the potential to 
enhance the assessment process by incorporating gamification elements. We designed a Gamified ORF progress- 
monitoring tool, KidSpeak. This paper describes the design of KidSpeak and discusses the application of KidSpeak 
to help teachers monitor students’ literacy development. 


Introduction 


Reading is recognized as a foundational skill: reading-success not only dictates school-success, but also 
improves self-esteem and enhances opportunities for higher education and future employment (Hudson, Lane, & 
Pullen, 2005). Oral reading fluency (ORF) has been identified as one of five essential components of reading by The 
National Reading Panel and National Institute of Child Health and Human Development (2000). 

However, the process of administering and scoring ORF tests is time-consuming since it requires 
individualized administration and for performance scores to be manually recorded. In addition, the reliability of 
scoring paper and pencil ORF tests is questionable since teachers generally cannot review student’s reading for 
errors and tests be re-scored. However, a computer-based assessment system could ameliorate these limitations and 
allow game elements, such as levels, points, and badges to improve learners’ motivation. In this manuscript, we 
KidSpeak, a gamified ORF progress-monitoring tool, designed to help teachers monitor ORF. 


Oral Reading Fluency 

ORF has a close relationship with comprehension; fluency helps students to make connections among the 
ideas in a passage and supports comprehension (Stanovich, 1991). Moreover, the results of ORF tests can help 
teachers to identify students at risk of reading failure and to make data-driven instructional decisions that improve 
students’ academic outcomes (McMaster, Shin, Espin, Jung, Wayman, & Deno, 2017). 

To assess ORF, teachers often administer assessments weekly. The student reads a brief passage while the 
teacher listens to the child read, and notes reading errors. The student’s score is calculated by counting the words 
read correctly per minute (WCPM) (e.g., Hudson, Lane, & Pullen, 2005) as research has shown WCPM to be an 
indicator of overall reading proficiency and is predictive of reading comprehension (e.g., Decker & Buggey, 2014; 
Fuchs, Fuchs, Hosp, & Jenkins, 2001). However, counting the number of WCPM is a time consuming and error 
prone process that could be improved by using computer technology. 


Progress Monitoring Systems 

A progress monitoring system is frequently administered set of measures used to determine whether the 
students are making satisfactory academic performance (Stecker, Fuchs, & Fuchs, 2008). Progress monitoring has 
been developed to measure mathematics (Ysseldyke & Tardrew, 2007) and literacy (Espin, Wallace, Lembke, 
Campbell, & Long, 2010; Shinn, & Shinn, 2002; FastBridge, 2019). In our system, we added gamification elements 
to an ORF progress monitoring measure. 


Gamification 


Gamification refers to the use of game elements, such as points, levels, and badges, to improve user 
experience and engagement (Deterding, Sicart, Nacke, O'Hara, & Dixon, 2011). Although gamification is not 
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associated with improved reading knowledge or skill, studies have shown that gamification can improve learner 
motivation (Brull & Finlayson, 2016) and motivation is associated with reading engagement, strategies, and success 
(e.g., Aarnoutse & Schellings, 2003; Fink & Samuels, 2007; Guthrie & Humenick, 2004; Guthrie & Wigfield, 
1997). Gamification is also appealing since it can be applied to learners with different learning styles, including 
auditory, visual, and kinesthetic (Pettit, McCoy, Kinney, & Schwartz, 2014). 


Design of KidSpeak 


In order to make ORF measures more efficient, informative, and reliable, we designed an ORF progress 
monitoring tool, KidSpeak, as part of a larger progress monitoring system known as AvenuePM 
(https://avenuepm.org). KidSpeak includes separated student, scoring, performance charts interfaces. 


KidSpeak: Student Interface 

The student interface was originally designed as an iPad app, since iPad are commonplace in schools in the 
US and the touch screen is easy for children to use. Recently, we developed a gamified version of KidSpeak, for use 
on mobile devices and computers (see Figure 1). The database includes 237 KidSpeak passages across 12 reading 
levels from mid-first to end of sixth grade. Each level is associated with a criterion score that constitutes success. 
Table 1 shows the success criteria for each level. Levels are indicated by images representing an animal hierarchy. 
The bar at the top of the page indicates the student’s current level. The student moves up a level by achieving 
success on 3 successive trials. Each success is indicated visually as a step toward the goal. In Figure 1, the current 
level is represented by a fish and the next level by a seahorse. 


KIDSPEAK 


Cle 


Back to Home 
Page 


Figure 1. Gamification feature of KidSpeak 


Table 1. Criteria for moving up a step 


Level 1 2 3 4 5 6 
Criterion 1 23 53 72 89 92 

Level 7 8 9 10 11 12 
Criterion | 107 112 123 127 139 140 


After starting a task, a text passage at the student’s reading level is shown on the screen (see Figure 2). 
After pressing the record button, the students are given one minute to read the passage aloud. A clock in the bottom 
right corner of the screen shows the time remaining. Upon completion, an audio file is uploaded to a server for 
scoring and analysis. 


384 


«BACK 


SATURDAY 


Saturday is the best day of the week. That is the day that you have free time. When 
Saturday comes I want to call my friends. They are just as happy about Saturday as I am. 
We start calling each other on the phone. We find out that one of us has a job to do. Or, we 
may have some place to go with our family. It’s hard finding time on Saturday to play at the 
same time. My parents make plans for me on Saturday. I don’t like it. All kids should have 


free time on Saturday! 


0:00 / 0:00 


Figure 2. Interface of KidSpeak test 


KidSpeak: Scoring Interface 

KidSpeak tests are scored in a custom scoring interface (see Figure 3). A scorer listens to a recording of a 
student reading and scores the test by selecting misspoken words in the interface. An ‘incorrect’ icon is placed over 
each misspoken or missed word. To complete the scoring process, the scorer selects the last word spoken by the 
student. Next, the scorer selects the Auto Correct button which marks all words up to the last identified word as 
correct. Finally, the software calculates the number of correct/incorrect words and adds this information to a student 
performance chart. 
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u| JENNY JONES Show Hints To Do List (5 Bug Report 


TOO MUCH RAIN 


VV VV VV VV VVV VY VV VV VC VV KVV VV VV 
We have too much rain. Winter is over. Now it is raining. We have very cold winters. We have snow for five months in the winter. 


X VV ¥ ¥ VC X¥VVV KX VY VC KC VC ¥- VK VVV VV VY 
Snow is fun when winter comes. Snow is not fun after five months. When spring comes the snow melts. | want to play outside 


Y ¥ KX VVV ¥ VC VV VY VV VC VV VVVVV VV VV 
when the snow melts. | can't play outside. It is raining. It rains almost every day. That is why | say we have too much rain. 


Use the spacebar to pause/ play the audio. Click/ double-click words to toggle Correct/ Incorrect/ Ungraded. 


| & | 0.00 0:51 


¥ x 
Correct 70 Incorrect 6 Total Words 76 Score 64 


Figure 3. KidSpeak scoring Interface 


KidSpeak: Performance Charts 

Performance charts are displayed at two levels referred to as macro and micro levels. Figure 4 shows a 
student performance chart. Each dot on the macro-level chart represents performance for a given week. Scores are 
averaged if a student completes multiple tasks in a week. Scores are displayed for up to 50 weeks, but the interface 
can be extended to show additional time periods. More detailed student information can be accessed by selecting 
one of the levels on the abscissa. 
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Drag the white bar to 
view data in different 


weeks 


Click on a level to view 
scores 


Vv 


Highest Le: 
28 © RADON O 


Weeks 


Drag the white bar to 
view data in different 


Hint 


Click on a data point to 


view, add, and delete 


notes. 


SCORES IN WEEK 245 


2019-05-01 
2019-05-02 
2019-05-02 
2019-05-02 
2019-05-02 


w 


BS 240 245 250 


Miufufoto 
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Figure 5. KidSpeak micro-level data display 
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Discussion 


ORF is accepted as a valid approach to monitor early reading development, but the processes of 
administering and scoring ORF tasks can be inefficient, time-consuming, and unreliable. This paper reports on the 
design of a gamified progress-monitoring tool, KidSpeak, which has potential to improve teacher efficiency and to 
provide teachers with information that can help identify students who are making inadequate reading progress. 

The AvenuePM website, including KidSpeak, is free for teachers and students. We plan to develop more 
passages and conduct further studies to investigate the effectiveness of the tool. The following research questions are 
planned: 

What are test-retest and alternate form reliability of the KidSpeak passages? 

How valid the KidSpeak measure? 

To what extent does KidSpeak save teachers when compared to traditional paper and pencil ORF tests? 
Does the KidSpeak data chart help teacher to identify students who are not meeting their ORF goals? 
To what extent does the gamification feature affect learner motivation? 
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Descriptors: instructional design, course design 


Queens’ Course Design Institute is an innovative course design program for faculty teaching face-to-face, 
hybrid, and online courses at Queens University of Charlotte. Offering three major innovations in comparison with 
course design institutes at other universities, Queens’ Course Design Institute synthesizes complementary aspects of 
the design approaches of Fink, Wiggins and McTighe, and Dick and Carey to empower faculty members to identify 
the key skills for which learners need particularly well-designed instruction. Faculty participants are guided through 
a backwards course design process focused on providing significant learning experiences through targeted 
application of a systematic design process. The implementation of the institute combines principles of project 
management, andragogy, social learning, and Bersin’s notion of “learning in the flow of work” to deliver innovative, 
inspiring workplace learning consistent with the 70-20-10 model of learning and development. 


Institutional Context, Needs Assessment, and Delivery Mode 


Queens University of Charlotte is a small, private, co-educational, master’s-level university in Charlotte. 
Queens’ mission is to provide transformative educational experiences that nurture intellectual curiosity, promote 
global understanding, encourage ethical living, and prepare individuals for purposeful and fulfilling lives. In 
carrying out this mission, Queens enrolls 2536 students and employs 131 full-time faculty, 173 adjunct faculty, 293 
staff members, and 286 student employees. As of 2018, Queens has as 9:1 student-faculty ratio and an average class 
size of 14. Its degree programs span liberal arts and professional programs in the humanities, social sciences, 
education, communication, health professions, and business. Its undergraduate programs include robust global 
learning and internship programs and a nationally distinctive general education curriculum that uses a series of 
learning communities to develop students’ ability to solve problems and address complex questions by drawing on 
multiple disciplinary perspectives (Queens University of Charlotte). 

Responding to the ramifications of this institutional scale, the Course Design Institute was designed to 
stress basic principles relevant to courses of various modalities and levels. The process used to design and develop 
the institute included needs assessment conducted through dialogue with various campus stakeholders, including 
faculty, staff, and administrators over seven months and review of pre-existing support in course design for all 
course modalities. As a result, the institute was created to support faculty in designing and redesigning courses 
taught in face-to-face, hybrid, and online modalities at the undergraduate and graduate levels. Faculty participate in 
the institute on a voluntary basis. 

Participants in the institute work on the design or redesign of one course that they will be teaching in the 
next fall and/or spring semester. The institute consists of four, day-long, in-person sessions held weekly in summer 
supported by an online learning platform used to share materials with faculty participants, support collaborative 
learning, and review and give feedback on faculty work. Holding the institute in June enables participants to 
continue applying design principles and concepts gradually during the remainder of the summer prior to fall or 
spring implementation of a new course design or redesign. Participants’ target courses illustrate variety in level, 
topic, modality, and discipline. Targeted courses include lower-division courses for majors, undergraduate research 
methods courses, and general education courses. They span face-to-face, hybrid, and online modalities. They 
address methodologies and topics in the humanities, arts, social sciences, natural sciences, and health professions. 


An Integrated Approach to Course Design 
Queens’ Course Design Institute integrates strong elements of three approaches to course design. The first 


key ingredient is the general concept of backwards design as illustrated in many instructional design models (Allen 
& Sites, 2012; Dick, Carey, & Carey, 2015; Cennamo & Kalk, 2019). Among K-12 and university teachers, the 
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phrase “backward design” or “backwards design” is most often associated with Wiggins and McTighe’s 
Understanding by Design. In particular, Wiggins and McTighe’s (2005, p. 18) three-stage version of backward 
design—1) identify desired results, 2) determine acceptable evidence, and 3) plan learning experiences and 
instruction—represents widespread principles of instructional design in a simple format that can be readily 
understood by instructors with no instructional design training. For the purpose of the Course Design Institute, 
however, a more elaborate representation of backwards design is used that reflects the institute’s integration of two 
of other approaches to course design. 

The second key ingredient is the emphasis on significant learning as elaborated and popularized at the 
university level through Dee Fink’s Creating Significant Learning Experiences. Significant learning can bring about 
lasting change that is important in learners’ lives after the course. Significant learning experiences in a university- 
level course can support learners in living enhanced individual lives, engaging in positive social interactions, 
pursuing informed and thoughtful participation in civic activities, and being prepared for work. In Fink’s taxonomy, 
significant learning is supported by a combination of foundational knowledge, application, integration, the human 
dimension, caring, and learning how to learn. Fink’s taxonomy assists instructors in shifting from a content-centered 
paradigm of college teaching to a learning-centered paradigm of college teaching. 

The third design approach integrated in the institute is from Dick and Carey’s instructional design model. 
While this model has been supplanted in various circles by constructivist, rapid, and agile approaches to 
instructional design during the past quarter-century, key components of Dick and Carey’s approach hold great value 
in a university setting. In particular, subordinate skills analysis as presented by Dick and Carey is a powerful tool for 
identifying a root cause for why past instruction has not produced the results desired. Subordinate skills analysis can 
expose challenging skills that novices lack and need instruction on but that instructors may have assumed, as 
experts, come naturally to learners. Additionally, Dick and Carey’s presentation integrates other useful techniques. 
These include Mager’s tripartite model for writing instructional objectives (conditions-behavior-criteria) and a five- 
part instructional strategy that blends the nine events of learning detailed in Gagne’s (1985) Conditions of Learning 
with Keller’s (2010) ARCS model of student motivation by addressing attention, relevance, confidence, and 
satisfaction. This synthesis supports the types of deep learning valued at Queens and similar institutions while 
ensuring that faculty members identify and address the subordinate skills that students need the most support in 
developing. 


Structure of Queens’ Course Design Institute 


The integration of strong elements from these three approaches to course design is carried out through a 
distribution of learning objectives across the four, weekly, in-person sessions in an order that largely mirrors the 
sequence of steps used in a backwards design process itself. As detailed in Figure 1, each weekly in-person session 
focuses on four to six learning objectives. This distribution of learning objectives reflects both an initial plan and 
adjustment of that plan during delivery in response to learners’ needs and their shaping of in-person instruction. The 
intended sequencing of the learning objectives was retained, but in practice, the first and third sessions addressed 
fewer learning objectives than planned. This resulted in delaying some objectives to the subsequent session, as 
reflected in Figure 1. 

The first session establishes a clear contrast between a common faculty approach to creating courses and a 
backwards design process as articulated through many instructional design models. After highlighting this contrast, 
the first session focuses on essential initial analytical steps found in many instructional design models: analyzing the 
context in which learning will occur and analyzing learner characteristics. The session then turns to Fink’s taxonomy 
of significant learning. Fink’s taxonomy is related to Wiggins and McTighe’s six facets of understanding and 
Bloom’s and Krathwohl’s taxonomies of the cognitive and affective domains. The session then prompts participants 
to clarify how their course will foster significant learning and articulate a major “end of course” performance that 
can be used to assess that significant learning (e.g., an individual research paper or poster presentation). Shifting to 
concepts drawn from the Dick and Carey text, participants then conduct a goal analysis by identifying the steps 
executed and key decision points addressed when carrying out that performance. 

The second session gives participants tools to dive deeper into their analysis and begin the process of 
articulating sound instructional strategies to help learners navigate the gap between their current performance ability 
and the level of performance they should have at the end of the course. Drawing on concepts from the Dick and 
Carey text, participants are guided in conducting a subordinate skills analysis (called “component skills analysis” in 
the institute) for one or two key steps in their goal analysis. After highlighting the differences between how novices 
and experts approach performing the same task, the session next guides participants in writing three-part learning 
objectives for some critical subordinate skills. Continuing to draw on relevant content from Dick and Carey’s 
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approach, the session highlights effective use of four types of tests: entry skills tests, pretests, practice tests, and 
posttests. The session guides participants in writing a five-part instructional strategy for a subordinate skill objective 
using Keller’s ARCS model of motivation. The session concludes with a prompt to pause and check for alignment 
between the subordinate skill, objective, and instructional strategy. Homework prompts participants to outline a five- 
part instructional strategy for two other subordinate skills that tend to be challenging for learners. In this process, 
participants revisit the learner analysis and context analysis to take into account learner motivation, goals, attitudes 
towards the subject, and previous relevant experience so that they can address relevance, confidence, and 
satisfaction from a learner perspective. 


Understand how a 
backwards design 
process differs from a 
common faculty process 
for creating a course 

Analyze the context in 
which learning will 
occur 

Analyze learner 
characteristics 

Clarify how your 
course will foster 
significant learning 

Articulate a major 
“end of course” 
performance to assess 
that learning 

Conduct a goal 
analysis based on that 
performance 


Conduct a component 
(subordinate) skills 
analysis 

Identify differences 
in how novices and 
experts perform the same 
task 

Write component 
skills objectives 

Distinguish the uses 
of four types of “tests” 

Outline a five-part 
instructional strategy for 
component skill 
objectives using the 
ARCS model of 
motivation 

Check for alignment 
between skills, 
objectives, and 
instructional strategy 


Design-in enhancement 
of learner motivation 

Design-in support for 
student metacognition 

Identify any major 
disconnect you may have 
introduced 

Remedy the 
disconnect(s) 


Sequence instruction 
effectively 

Map instructional 
strategy to the context 

Select suitable 
instructional materials 
and deliver them 
effectively 

Schedule instruction 
into semester structure, 
consistent with course 
modality 

Create syllabus 

Develop plan for 
completing re/design 
project 


Figure 1. Learning objectives as distributed through the four in-person sessions of Queens’ Course Design Institute. 


The third session delves deeper into student metacognition and learner motivation, as stressed in Fink’s and 
Dick and Carey’s approaches, respectively. This session responds to both a pervasive discourse on metacognition at 
Queens as established in the general education program and discourse on student motivation stemming from ample 
experiences supporting a diverse population of traditional undergraduates and working adult learners at the 
undergraduate and graduate levels. The session showcases three related models of motivation and practical ways to 
support students in developing metacognitive skills and engaging in self-regulated learning. The session then 
prompts participants to identify any major disconnect or misalignment between objectives, assessments, and 
learning activities that they may have introduced in their course design work to date. 

The fourth and final session focuses on planning the effective implementation of the decisions made in 
previous sessions. This includes sequencing instruction effectively, mapping an instructional strategy to a particular 
instructional context including course modality, awareness of key considerations when selecting and delivering 
suitable instructional materials from a wide range of options, scheduling instruction within a semester structure 
consistent with the course modality, and creating a syllabus. Essential to this process is developing a plan for 
completing the course design or redesign project in relation to the timeline for delivering the course the following 
fall or spring semester. Accordingly, the homework includes a project completion timeline and a progress report due 


in the first half of August. 
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Guiding Design Principles 


The design and development of the Course Design Institute were guided by both practical and theoretical 
principles. One set of principles was nine practically oriented grounding principles developed from sustained study 
of adult and organizational learning and tailored to the needs, structure, and organizational culture at Queens: 


Give learners choices and autonomy 

Use images and interaction for memorable learning 

Leverage colleagues’ knowledge and experience 

Ask questions to promote reflection and shared responsibility for learning 
Deliver good design elastically 

Model sound practices 

Foster mutual respect and trust 

Promote learning opportunities in relationships as much as “training” 
Advocate the connection of learning to performance management 


SOAS ie Bee 


These grounding principles state in concise and practical terms guidance developed from adult learning, including 
andragogy, social learning, and Josh Bersin’s (2018, 2019) notion of “learning in the flow of work.” 

Of these, andragogy most significantly influenced the design of the institute. Approaching the participants 
as self-directed learners with strong internal motivation oriented towards applying concepts to solve specific 
problems in the immediate future shaped learner activities and instructional materials. Knowles’ (2005) assumptions 
of andragogy influenced the institute’s design through the emphasis placed on relating information to participants’ 
abilities to solve real problems in their immediate work. Responding to participants’ likely orientation to problems, 
abstract or theoretical information was chunked into manageable segments and followed promptly by application 
activities. Responding to adult participants’ likely internal motivation and desire for self-direction, the institute 
included ample opportunity for participants to make choices among curated activities, instructional materials, and 
homework prompts. For example, homework after the third session included three options. Participants could 1) 
remedy a major misalignment between learning objectives, assessments and instructional activities by using 
techniques and concepts from the first three sessions, 2) continue applying goal analysis, subordinate skills analysis, 
the five-part instructional strategy, and models of motivation to other subordinate skills that students have struggled 
with in the past, or 3) formulate reasonable plans to support the development of self-regulated learning by 
identifying a specific metacognitive skill, writing a three-part learning objective for it, and sketching a five-part 
instructional strategy that supports the development of the metacognitive skill. 

Social learning also influenced the design of the institute. Application activities were often followed by 
sharing and group discussion as one outlet for social learning (Horton, 2012). Allowing time for participants to share 
anecdotes of relevant experiences with each other and respond to each other rather than having exchanges with the 
primary session facilitator turned group discussion into peer learning. Due to aspects of organizational culture, peer 
learning readily extended into peer mentoring as participants shared additional strategies and advice based on 
teaching experiences at Queens. As a result, for segments of several sessions, one or more participants became a 
dominant voice rather than the institute’s primary facilitator. This chain of events flowed from grounding principles 
3 and 8. 

While the institute was carefully designed it was also flexibly run, as encapsulated in guiding principle 5. 
Participants’ interest in social learning and peer mentoring was so strong during the first and third sessions that 
planned content and activities had to be pushed into the second and fourth session. Homework planned to follow the 
first session had to be scrapped entirely. 

The result was a form of workplace learning more consistent with a 70-20-10 model for learning and 
development. This widely promoted model asserts that 70% of learning in the workplace occurs informally during 
job-related experiences, 20% occurs through developmental relationships with others, and only 10% occurs through 
formal training. While the source of the percentages foregrounded in this familiar model has been questioned and 
the model’s positioning as fact has been debunked (Jefferson & Pollock, 2014; Clardy, 2018), the general idea 
encapsulated in this model remains a valuable counterweight to the traditional emphasis on training in human 
resources and instructional design arenas. It reminds us that peer learning and curated just-in-time resources 
available for learners to pull on demand, along with many other forms of learning support, are valuable ingredients 
in a larger learning environment that supports both individual and organizational learning (Lombardozzi, 2015). The 
flexible delivery of the institute and accommodation of participants’ habitual transformation of training or 
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conversation into peer learning and peer mentoring responded to situational factors at the root of Bersin’s (2018, 
2019) notion of learning in the flow of work and at the point of need. 

Finally, essential to the design of the institute was attention to project management principles and the need 
to equip participants with the project management resources and skills to ensure completion of sound course design 
after the institute and before faculty implementation of a new course design or redesign (Rothwell, Benscoter, King, 
& King, 2015). For this reason, a significant portion of the fourth session and the subsequent homework was 
dedicated to basic aspects of project management, including chunking work into manageable units, specifying 
resources needed for project completion, and constructing a feasible timeline with relevant milestones. Such 
timelines typically included the construction of a course schedule and syllabus after the institute’s final session but 
by August. 


Key Innovations 


The innovation of Queens’ Course Design Institute stems in large part from how it differs from course 
design institutes offered at other universities. Most course design institutes focus on the design of face-to-face 
courses, draw on one design text, and are completed in three to five consecutive days within one week. Queens’ 
Course Design Institute innovates in each of these respects. 

First, Queens’ institute is structured as a series of four day-long sessions occurring weekly throughout one 
month. This enables faculty participants to apply and deepen their understanding of key concepts and techniques as 
they work on their design projects between weekly sessions. Additional time between in-person sessions can be used 
to receive and provide feedback. It also provides time to work as needed with relevant campus resources and 
partners and make connections with relevant programs, such as the General Education program and the Hayworth 
Center for Online Learning. This contrasts with the compressed schedules commonly used in course design 
institutes (see Figure 2). While offering course design content over three to five successive days may be convenient 
for facilitators and participants, it does not provide ample time for incorporating, reflecting on, and applying 
concepts. This time is especially important given that instructional design concepts are interrelated and build on each 
other. 


Main Topic: Student Learning Main Topic: Assessments Main Topic: Teaching and 
Objectives Aligning learning objectives and Learning Activities 
Course constraints and challenges assessments Passive vs. active learning 
Syllabus template Student-centered assessments Active learning strategies and tools 
Draft and revise learning objectives Types of feedback Getting students to success on an 
Draft a course description Draft and revise draft an assessment assignment 
Begin organizing course schedule Add to course schedule Refine learning activities 

Refine course schedule 

Begin to assemble full syllabus 


Figure 2. Three-day Course Design Institute schedule summary (derived from Duke University, 2019). 


Second, Queens’ Course Design Institute synthesizes strong, complementary elements of three approaches 
to course design, whereas most course design institutes emphasize one approach: Fink’s Creating Significant 
Learning Experiences (see Figure 3). The ramifications of this difference become clear when reviewing Fink’s text 
with a broader instructional design perspective. Fink’s approach begins with familiar steps in a backward design 
process and a check for alignment or “integration” between learning goals, feedback and assessment procedures, and 
teaching and learning activities. However, his stepwise approach does not prompt instructors to identify the 
subordinate or component skills that learners must have to achieve the significant learning goals as displayed in the 
aligned assessments. This step is critical given the complexity of skills captured in several areas of Fink’s taxonomy, 
including application, integration, and the human dimension, such as leadership and teamwork. It becomes more 
critical when the differences between novices and experts are taken into consideration (Ambrose, Bridges, DiPietro, 
Lovett, & Norman, 2010). When those following Fink’s steps move into creating a thematic structure for the course 
(i.e., selecting and sequencing the most important concepts, issues, topics, or themes) and selecting or creating a 
teaching strategy (i.e., a combination of learning activities that have been arranged in a particular sequence), there is 
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a risk that instructors do not incorporate instructional modifications necessary to support novices in learning or 
synthesizing essential subordinate skills as needed to achieve the learning goals for the course. Moreover, the 
resulting thematic structure and teaching strategy may be overly driven by inaccurate estimates of time needed or by 
repetition of activities in particular patterns, as in the illustrations of problem-based learning. 


from learning goals 
and objectives 
Defining what you 
most want students 
to get out of your 
class 

Individual work and 
consultations 
Determining your 
learning goals and 
objectives 

Create a new 
learner-centered 
course description 


feedback on student 
learning 
Developing learning 
assessments 
Individual work and 
consultations 
Optional mini- 
workshop on 
technology 

Refine your 
learning objectives 
Determine overall 
assessment strategy 
Develop 1-2 
specific assessments 


activities 
Developing learning 
activities 

Individual work and 
consultations 
Optional mini- 
workshop on 
technology 

Refine assessments 
Determine overall 
instructional 
strategy 

Develop 1-2 
specific activities 


CDI Overview Design Feedback and Creating the Exchange syllabi 
Understanding Principles of assessing student schedule Implementing the 
student motivation gauging student learning Feedback and design 

Principles of course _ learning Principles of active grading Turn in paper copy 
design Exploring ways to learning Individual work and _ of near-final or final 
Backward designing assess and gain Exploring learning consultations syllabus 


Exchange syllabi 
Individual work and 
consultations 
Refine learning 
activities 

Define grading 
scheme 

Determine tentative 
schedule 

Complete syllabus 


Figure 3. Schedule for a five-day Course Design Institute based on Fink’s Creating Significant Learning 
Experiences (derived from George Washington University, 2016). 


So rather than drilling down through the hierarchical layers of a subordinate skills analysis to expose and 
address essential skills that have gone (or are at risk to go) untaught or undeveloped, course design institutes often 
place emphasis on Fink’s shift from a content-centered paradigm of college teaching to a learning-centered 
paradigm of college teaching. And participants’ success in making this shift is often captured through an assessment 
of the syllabus using a coordinated syllabus rubric (Bromley, 2015; Palmer, Streifer, & Williams-Duncan, 2016; 
Palmer, Bach, & Streifer, 2017). The course design may even be equated with or reduced to the syllabus: as 
Steinhardt (2015) explained the impact of a course design institute on faculty at George Washington University, “By 
the end of the week, they’ve created a new or redesigned project: a new or redesigned course syllabus, one that Dr. 
[Michael] Palmer explains is “learning-focused” rather than “content-focused.” Or as Bromley’s (2015) title 
misleadingly suggests in contradiction of Fink’s own text, “Building a better course starts with the syllabus.” 

A tush to create and measure a syllabus by the end of a three- or five-day institute may entail premature 
decision-making in other critical areas of the course design or redesign. One is the selection of instructional 
resources (e.g., readings, audio-visual materials). As these are normally specified in a syllabus, there may not have 
been enough time to select them with ample attention to learner analysis and motivation. Another is the relative 
duration allotted for instructional activities, practice, feedback, and assessment of learning aligned with various 
learning objectives. If given more time before the development of the syllabus and guidance in subordinate skills 
analysis, instructors would likely alter the amount of time allocated for the development and assessment of learning 
objectives that vary in complexity. Moreover, they would likely allow additional time for learners to have guidance, 
practice, and feedback in synthesizing subordinate skills as needed for various forms of significant learning. 

Finally, Queens’ Course Design Institute stresses key design steps and principles relevant to all course 
modalities. In contrast, many course design institutes are designed to support faculty creating and teaching face-to- 
face courses. Others are geared primarily towards online or hybrid courses. Supporting faculty teaching in all 
modalities was critical given Queens’ size and the mixture of modalities handled by any one faculty member. For 
example, a faculty member teaching exclusively face-to-face courses during fall and spring semesters will often 
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teach in an online modality in summer to support students who have returned to live elsewhere in the country. 
Moreover, faculty may be responsible for teaching the same course in more than one modality. This cross-modality 
approach to the institute is further supported by the fact that many of the same fundamental criteria are important 
when evaluating teaching or course design in online, hybrid, or face-to-face modalities. Examples of such criteria 
include clarity about learning outcomes or objectives, student engagement, and the alignment of assessments with 
learning objectives (Baldwin, Ching, & Hsu, 2018; Central Piedmont Community College 2019; Lohman, 2019). 
Therefore, while guiding participants to draw on modality-specific resources in their work, Queens’ Course Design 
Institute helps raise faculty awareness of and capacity for strong course design across modalities, something that is 
especially important in institutional settings with an unfavorable ratio of instructional design staff to faculty or a 
modest headcount of instructional designers. 


Designer Observations and Participant Feedback 


Both my observations as the designer, developer, and primary facilitator of the institute and participants’ 
feedback confirm the value of the innovations in the institute and the principles used in developing and delivering it. 
I had the unique opportunity to deliver Queens’ Course Design Institute while also working with and observing 
graduate students in an instructional design course at another university. Learning instructional design techniques 
and processes—especially as a novice—can be challenging for faculty and other learners alike. The temptation when 
hitting an unfamiliar step or a roadblock is to use old habits and skip critical steps. For example, when first asked to 
write an instructional strategy (for oneself to later develop and implement), sheer habit and comfort may lead a 
faculty member to begin writing instructions for and to students instead. Similarly, when asked for the first time to 
write the learning objectives to be developed in each portion of an employee orientation program, a human resources 
professional experienced in running such programs may prematurely write a time-based agenda for the program out 
of familiarity, comfort, and habit. 

One of the most significant takeaways from facilitating the institute came from seeing participants’ 
reactions to component skills analysis (Dick and Carey’s subordinate skills analysis). Despite having a strong belief 
in its value, based on approximately 13 years of experience implementing such principles in university instructional 
settings, I was concerned that faculty participants in the institute might perceive this step as overwhelming, tedious, 
or of unclear benefit. On the contrary, participants quickly saw its value and readily connected it to their own past 
experiences as instructors. They suddenly understood why students had not performed as they had hoped on key 
learning assessments in previous courses. Several participants’ lightbulb moments came from applying component 
skills analysis to their own courses and realizing exactly where they had assumed students could just do something 
that they as experts could do and where they had not stopped to pause and provide needed instruction and practice 
(e.g., the skill of identifying effective keywords for a library or database search). This reaction came in large part 
from translating Dick and Carey’s techniques into examples relevant to the learners. 

Participants’ feedback through an anonymous end-of-institute survey clarified the most valuable aspects of 
the institute. When asked “What are the top 3 things you learned in this institute?” participants most frequently 
highlighted component skills analysis, followed by backwards design and supporting students’ development of 
metacognitive skills. Articulating the value of a component skills analysis for recognizing the needs of novice 
learners, one participant noted that a most important outcome was “Realizing there are skills so inherent in my 
process that I've never stopped to consider them as individual skills, let alone teachable moments.” Another 
highlighted how this particular step played a key role in the backwards design process when identifying a major 
takeaway as “Focusing on skill components rather than jumping into instructional strategies.” Such specific 
feedback confirms the value of integrating elements of Dick and Carey’s instructional design approach with Fink’s 
in university settings. 

Participants’ clear plans for applying knowledge gained in the institute confirm the value of integrating 
backwards, significant learning, and systematic approaches to course design as outlined in the texts of Wiggins and 
McTighe, Fink, and Dick and Carey. Participants’ plans included resequencing the target course, creating a different 
timeline or schedule for the course, and redesigning instructional materials, such as PowerPoint presentations, 
lectures, and in-class activities. Both the component skills analysis and five-part instructional strategies taught by 
drawing from the Dick and Carey text fueled these participants’ plans for how they would spend instructional time 
in their courses. When asked “What are some specific ways that you will apply what you learned in this institute?” 
one participant explained, “I will go back and really structure instructional time in the classroom focusing on the 
specific skill components I need to prioritize. Think through the 5 step process that will help me be more intentional 
and planful :) rather than simply implementing an instructional strategy because I learned about it. The 5 step 
process allows me to be more aware of why I am doing something and when (and how).” 
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This positive feedback was confirmed as participants began implementing their revised course designs. 
Participants continued to cite key design techniques from the institute during the semester and also used those 
techniques to shape their instruction in other courses. As a result, the learning begun in the institute was sustained 
and further developed in the course of faculty members’ teaching activities and in ordinary interactions with 
colleagues. This continuation of learning begun in the institute aligns with the basic premises of Bersin’s “learning 
in the flow of work” and the 70-20-10 model of learning. 

The success of the institute stems from many factors, including the self-selection of the participants, the 
timing of the institute, the collegiality and mutual respect fostered among all involved, and the commitment to 
teaching that pervades daily interactions at Queens. However, beyond these factors is the institute’s unique 
mediation of faculty and instructional design perspectives. This mediation is achieved by showcasing key 
instructional design techniques through specific examples created and chosen for their relevance for a faculty 
audience. It is also achieved by modifying terminology for the audience at hand. Thus “subordinate skills analysis” 
is presented as “component skills analysis,” to stress the ultimate goal of building up learners’ skills towards a 
desired performance, which is aligned with faculty experience in a class setting, rather than the intermediate goal of 
decomposing skills by working downwards through a hierarchical diagram, aligned with experience in an 
instructional design role. And despite the comprehensiveness of the golf examples diagrammed in Dick and Carey’s 
text and their utility for showing how to address verbal, intellectual, motor, and attitudinal skills, the concept of 
subordinate skills analysis is illustrated in the institute through an example more immediately understood and more 
likely to be seen as relevant by the learners in the institute: creating a research paper. 


Conclusion 


Reflecting on Queens’ Course Design Institute highlights additional opportunities for the integration of 
perspectives on course design common among faculty and faculty developers on the one hand, and instructional 
designers on the other hand. The heavy reliance on Fink’s approach in many course design institutes is readily 
explained by the predominant role of faculty developers in creating, designing, and delivering course design 
institutes, rather than individuals with significant instructional design background. In the case of Queens’ Course 
Design Institute, both faculty and instructional design perspectives could be integrated and mediated by one person 
who has applied instructional design techniques in university-level instruction while serving as a faculty member for 
over a decade. At larger institutions, a collaborative approach to the design of such institutes can enrich the range of 
skills developed in faculty, foster greater mutual awareness of the complementarity of these perspectives, and assist 
instructional designers in being prepared to collaborate efficiently with faculty in both individual and group settings. 
Such a collaborative approach to a course design institute may be particularly valuable at institutions in which 
individual faculty-instructional designer partnerships are not sustainable or scalable. 

Unfavorable instructional designer-faculty ratios and differences in experience, vocabulary, and priorities 
can contribute to obstacles and frustration felt by instructional designers and faculty alike. These factors can hinder 
communication and collaboration between instructional designers and faculty. Queens’ Course Design Institute is an 
effective and innovative model for overcoming these obstacles and bridging these gaps. For instructional designers, 
the institute highlights the importance of using terminology in a way that others can readily understand, which may 
involve selecting carefully from or modifying existing terminology. Moreover, it demonstrates the importance of 
adapting or expanding fundamental concepts, such as learning objectives, in alignment with the values of a 
particular institution. It also underscores the need to be aware that some texts and basic concepts may not be widely 
understood among faculty. For faculty, the institute demonstrates the importance of being open to learning new 
techniques and adding them to one’s toolbox of techniques for use in designing and delivering learning experiences. 
For faculty developers, the institute offers an invitation to explore course design resources beyond Fink’s widely 
used text. In particular, the institute makes clear that elements of Dick and Carey’s approach still hold relevance and 
value in university settings, despite the rise of other valuable models and approaches to instructional design. 
Queens’ Course Design Institute highlights important elements that can be selected from other instructional design 
and course design texts beyond Fink’s and integrated into extended learning opportunities for faculty. In doing so, it 
offers a helpful and inspiring example to instructional designers and faculty that can be used to understand and 
address common obstacles hindering the feasible application of sound course design principles at the university 
level. 
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Abstract 

This cross-case analysis of building capacity for the effective adaptation of professional development 
content for educators across different cultural contexts was an international collaboration conducted between an 
instructional design team, two cultural consultants, and two international educators. Capacity occurs at three 
levels—individuals, work environment, and institutions (Malyan & Jindal, 2014). This cross-case analysis is divided 
into two phases, and delves into the first level: the individual. Phase One (P1) of this research involved a team of 
U.S. instructional designers and a Ukrainian educator introducing Ukrainian graduate students to the world of online 
learning. Phase Two (P2) expanded upon P1 with the addition of two cultural consultants to the team. Cultural 
consultants are individuals with an extensive knowledge of their own culture and community, and understand how 
services can be best presented to meet the needs of that community. P2 is an ongoing collaborative initiative 
between the Innovation in Learning Center (ILC) at the University of South Alabama (USA), Ivano-Frankivsk 
National Technical University of Oil and Gas IFNTUOG), and Universidad Pedagogica Nacional Francisco 
Morazan (UPNFM). 

The purpose of this cross-case analysis is to provide insight in building capacity. Capacity occurs at three 
levels—individuals, work environment, and institutions (Malyan & Jindal, 2014). This cross-case analysis delves 
into the first level, the individual, for the effective adaptation of professional development content for educators by 
integrating technology and learning strategies across different cultural contexts. 

Building capacity refers to the effort put towards improving the abilities, skills, and expertise of a school or 
individual to accomplish or execute specific goals (Wing, 2004). It is the process to assist a school or college’s 
ability in accomplishing its mission by combining elements and resources within an institution to benefit all 
stakeholders. The purpose of building capacity is to create and implement programs that will allow individuals to 
grow their own knowledge, skills, and abilities. For capacity building to be sustainable and remain effective, all 
stakeholders must invest in the improvement of the institution through collegial efforts. Change agents become 
strong leaders who can exemplify their expert knowledge to those around them, encouraging learner-centered 
approaches with performance-based settings that incorporate varied pedagogies and techniques that promote student 
learning and success. In order to begin capacity building, we divided this project into three different phases. Phase 


399 


One (P1) involved establishing rapport, relationships, and a pilot study of the online environment. Phase Two (P2) 
involves revisions of the data obtained in P1, along with additions to the research team. 

Completed in May 2018, P1 was a collaboration between an educator at the Ivano-Frankivsk National 
Technical University of Oil and Gas (IFNTUOG) in the former Soviet Republic of Ukraine and a team of 
instructional designers at the Innovation in Learning Center (ILC) at the University of South Alabama (USA). P2 is 
an ongoing collaboration between the ILC and IFNTUOG, while expanding that partnership with an instructor at 
Universidad Pedagogica Nacional Francisco Morazan (UPNFM) in La Ceiba, Honduras. Working with IFNTUOG 
and UPNFM, both educators in Ukraine and Honduras were faced with the overlapping themes of little support from 
local administration and had few resources at their disposal despite originating from two completely different 
cultures and geographical regions. This research was conducted as a collaboration to share faculty development 
content and provide access to educational technologies. It serves to establish the aptitudes, resources, relationships, 
and its facilitating conditions to effectively build academic capacity. 


Phase One 


Many international students come to the United States with little to no online learning experience. Online 
instruction can be different from what international students are accustomed to in their home countries. Adjustments 
associated with learning in this format could put global students at a disadvantage. P1 of this research involved an 
instructor at IFNTUOG and two instructional designers. This phase involved introducing Ukrainian graduate 
students to the world of online learning. Prior to introducing the students to their first online course, the instructor 
was trained and certified to teach in the Learning Management System (LMS). 

P1 indicated that success and sustainability are contingent upon the students, as they are the end consumers 
of this educational experience. Phase One data were collected through a series of video interviews and qualitative 
questionnaires. It revealed linguistic, social, and academic barriers the Ukrainian students faced as inexperienced 
online learners. An interview conducted with the instructor revealed evidence of bureaucratic resistance to change of 
the status quo due to lack of training and development of competencies. This response was to be expected as much 
of the literature has indicated these challenges were a particular problem (Blayone et al., 2018). 

The researchers found that implementing new approaches to learning content had a positive effect on 
students; however, the push from new educators who wish to expose their students to experiences that help them 
become more competitive in the global market, while also inspiring their administration to do the same, has proven 
to be a difficult task. Their efforts to serve as a change agent and transition into the second level of capacity 
building, bringing their colleagues forward in utilizing new learning technologies and strategies, and changing the 
old paradigmatic mindset of teaching and learning, has not gained much traction in terms of faculty development 
(Fink, 2013). 


Phase Two 


Cultural Consultants 

P2 expanded upon P1 with the addition of two cultural consultants. Cultural consultants are individuals 
with an extensive knowledge of their own culture and community, and understand how services can be best 
presented to meet the needs of that community. These individuals provide region-specific socio-cultural and 
linguistic knowledge, information, and advice that could only be acquired by living in an area or studying it for 
several years (IOR, 2018). Thus, the Ukrainian and Honduran team members served as cultural consultants and 
worked with an international partner in their home country. 

Linguistic and academic differences can be obstacles when developing, designing, and delivering content 
as the international partners differ greatly in academic language, interpersonal communications, and teaching 
practices from the ILC instructional design team. New technologies in teaching and learning have allowed for more 
impactful transnational relationships that result in frequent interaction and deeper engagement between international 
partners and the instructional design professionals who hold essential knowledge of specific academic cultures. The 
cultural consultants are natives of Ukraine and Honduras, and have both worked extensively in education. Their 
expertise provides much needed insight and input for the U.S. instructional designers to provide support that is 
conducive to the linguistic, academic, and cultural environment of the institution. As cultural consultants, they both 
served a formal and informal role in the professional development of their international counterparts. The 
international partners benefit from the working relationship with a cultural consultant who can expertly navigate the 
socio-cultural differences and serve as a mediator; therefore, bridging the gap between innovative technological 
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tools and instructional strategies used in the United States to the unique cultural needs and contexts of the 
international partner’s own culture. 

The cultural consultants worked with international partners to integrate content developed by the U.S. 
instructional design team by providing guidance and adapting instructional content to be culturally relevant to the 
faculty development needs of their respective countries. Utilizing cultural consultants will provide a sense of 
connectedness and belongingness which facilitates a sense of relatedness to their international counterparts. 
Evidence has shown that individuals are more likely to participate in actions valued by others to whom they feel a 
connection (Ryan & Deci, 2000). As a member of the target culture, establishing relationships between the cultural 
consultant and the international partner is essential to connecting to the academic endeavor. The researchers utilized 
cultural consultancy as a method to provide global guidance to ensure a seamless transition in overcoming existing 
cultural, technological, and academic barriers with the participants. The personal familiarity between the cultural 
consultants and international educators narrows the gaps of the cultural-specific needs and goals of the students 
involved in the research to sufficiently begin creating cultural relevance in an academic capacity. Both international 
educators proactively reaching out to create sustainable relationships through international collaboration serves as 
groundwork to inspire educators and students through global inclusivity. 


Implementing Phase Two 


P2 is an ongoing collaborative initiative between the ILC and IFNTUOG, while expanding this partnership 
to include an instructor at Universidad Pedagogica Nacional Francisco Morazan (UPNFM) in La Ceiba, Honduras. 
Academic reassessment requires partners embarking on a joint venture of exploring the quality of professional 
development, leading to mutual transformation of teaching effectiveness. As stated previously, this begins at the 
individual level of building capacity. To address the gap, this phase was designed as a cross-case study to examine 
the perceptions of international faculty from two different countries with distinct social cultures and similar 
academic cultures. 

This part of the research involved the work of six individuals: two instructional designers from the United 
States; two academic professionals representing two global partner sites; and two cultural consultants acting as 
liaison within the collaboration. The Ukrainian and Honduran partners share the desire to enhance the educational 
practices at their respective institutions to provide innovative and meaningful learning experiences for their students. 
Qualitative data from P1 was analyzed through open coding (Glaser and Strauss, 1967) through interviews held 
periodically throughout the course. At the end of each interview, data were compared to the information gathered 
previously. After the completion of the final session of the course, all P1 data were collectively analyzed. Themes 
that emerged from the existing literature, along with the data collected from P1, indicated a gap existed in 
institutions regarding building academic capacity for their faculty. P2 of the study addresses this gap, using 
multicultural perspectives to guide the process of transnational educational partnerships. Some of the challenges to 
e-learning P2 will be addressing are (Blayone, et al., 2017): 


Low-level digital readiness for online learning of the students, teachers, and administrators 
Faculty and administrators’ lack of training to guide students in the world of online learning 
Unreliable internet access and availability 

Lack of experience in self-guided and self-motivated learner-centered curriculums 


Both Ukraine and Honduras face these challenges. The processes of P2 will be successful if the partnership shows 
outcomes of increased capacity, with changes in attitudes and behavior that indicate that the future relationship will 
lead to sustainable, long-term connections that benefit a// stakeholders. 

Three questions will be addressed in this study: 

e To what extent do the international partners feel competent to integrate the instructional practices 
introduced to them from the intentional, ongoing, and systemic professional development support 
partnership with regards to building capacity? 

e To what extent are the students of the international partners impacted by the changes in teaching practices 
made as a result of the partnership? 

e To what extent does the use of cultural consultants impact the cultural relevance of the outcomes of the 
transnational partnership? 
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International Partners’ Sense of Competence 

With the robust development of instructional technology, education in a globalized world has been 
challenged to quickly adapt and offer learning models, which would diminish time, spatial, social, and cultural 
boundaries of the traditional classroom (Vonderwell, Liang, & Alderman, 2007). These circumstances present the 
potential to embrace a broader diversity of learners. The effective implementation of change in the delivery of 
learning places greater responsibility on the instructors, who have to create and facilitate effective online learning 
modes, inclusive environments, and design curricula that respond to competitive professional market’s requirements. 

Data collected in P| indicated that the level of instructor pedagogical and technological readiness had a 
negative influence on instructor satisfaction for pre-course delivery for the Ukrainian partner. She stated that the 
lack of technological expertise and understanding of online delivery was a source of anxiety and resulted in lower 
confidence in her ability to successfully teach in an online format. After the second online experience within the 
course, anxiety levels decreased, and she reported an increase in the level of satisfaction with the collaboration. Each 
learning experience throughout the course concluded with a debrief through online interaction via web conferencing 
tools. The debrief included feedback on delivery of the course content using the tools within the learning 
management system and suggestions of additional instructional strategies to utilize for the next online session. The 
partner gained certification for online course development through the ILC. She successfully completed the same e- 
learning training course that instructors at USA must complete before teaching online, with coaching from the ILC 
e-learning specialist. At the end of P1 of the online experience, the Ukrainian partner reported e-readiness, but 
identified a deficit in pedagogical readiness items. She wanted to seek to academic capacity and continue building 
global connections, however realized that international educational settings did not conform to the traditional 
Ukrainian educational setting. To avoid making assumptions about what will and will not work in this international 
collaboration, the instructional design team decided to seek the assistance of a cultural consultant. 


Students of International Partners 

Education in the 21st century requires students to develop their skills to become self-directed learners and 
develop into self-growers of constructing their own knowledge through self-efficacy and intrinsic motivation. As 
international partners reach the level of academic capacity that allows them to develop learning experiences that 
promote a sense of relatedness, measured in terms of teacher-student relationships, the sense of belongingness and 
connectedness amongst students increases (Ryan & Deci, 2000). The learning experiences that satisfy the basic need 
of relatedness have been linked to self-efficacy, engagement, interest in course content, higher scores, and student 
retention (Beachboard, Beachboard, Li, and Adkison, 2011; Furrer & Skinner, 2003; Inkelas & Weisman, 2003; 
Ryan and Deci, 2000) The purpose of education becomes more about the intrinsic value of learning rather than the 
consumption of knowledge (Burke, Lawrence, El-Sayed, & Apple, 2009). This is a paradigm shift from the 
conventional educational customs from the Ukrainian and Honduran contexts. This conventional method of 
education is also commonly shared across other cultures as well. 

Common themes were found through the online interviews conducted with the students, they expressed the 
importance of learning in an online environment; they were in different and usually low level of digital readiness for 
online-learning (Synytsya, K., & Manako, A, 2010). They felt challenged, because of this novel experience and also 
the internet access and availability. Despite the technological challenges students faced, we had our partners 
accommodate lab time use to ensure that they would have access to the internet and were able to complete the online 
activities, and also hold synchronous meetings with the cultural consultants. 


Next Steps 


The goal of this ongoing project is to infer technological and logistical feasibility, sustainability of 
established relationships with an outcome of increased academic capacity, and the development of culturally 
relevant content. After evaluating and completing P1, the researchers have determined that P2 is a feasible endeavor. 
The research has demonstrated itself to be effective, able to be implemented in a real-world setting, and serve as a 
model of a worthwhile sustainable approach to building academic capacity with cultural relevance. Feasibility was 
determined through feedback in P1 in relation to the students’ satisfaction with the setting of the course, learning 
content, and support of instructional designers. Feasibility was also determined by the barriers encountered during 
the project, technological implications, and organizational support. After uncovering the gaps from P1, we 
developed a strategy to fill those gaps with additional resources needed to successfully begin P2: creating academic 
capacity at the first level with the individual. 

The sustainability in progressing through all three levels of academic capacity is a long term goal of our 
partnership. This goal lies in providing the tools and support our partners need to continue building capacity in their 
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work environment and institutions, along with the opportunities we can facilitate for the Ukranian and Honduran 
institutions to collaborate with one another in the future. Some critical factors in sustainability were discovered in 
the initial phase of the project. The difficulty in obtaining support from the administration and institution has not 
garnered much attention in the past for the international partners. However, the passion for improvement from both 
international educators has presented them as community leaders committed to achieving academic capacity. To 
counteract the threats to sustainability, the next step for this project to achieve the milestone levels of building 
academic capacity is through varied strategies of ownership. As the first level of capacity is established with each 
international partner, the second level begins. As the partners invite colleagues from their organizations to join their 
efforts of professional development, with the aid of the cultural consultants and the instructional design team, this 
initiates the transfer of ownership from individual to the work environment at the second level of academic capacity. 
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This paper describes a research project designed to investigate major differences in accessibility to learning 
technologies in three of the world’s largest countries in terms of population: United States, People’s Republic of 
China, and Federal Republic of Nigeria. Each of these countries has unique social, economic, and political 
conditions that add value to the comparison conducted. As the researchers believe libraries serve an important role 
in the education of the public, both accessibility in school/university environments’ and within public, school, and 
academic libraries’ technologies were examined. 


Research Problem 


Little is known about the current state of accessibility for education on an International scale. Many studies 
that examine accessibility focus on a single institution or nation, lacking valuable comparative case study that 
informs readers and education professionals as to what practical measures can be taken to improve accessibility. By 
examining what some countries do right as well as identifying their weaknesses, professionals from these nations 
can work together to mutually support the improvement in accessibility. By identifying potential barriers to the 
diffusion and adoption of accessibility policy, professionals may work to ameliorate these barriers and create more 
equitable experiences for all learners. 


Theoretical Lens for Analysis of Discrepancies in Accessibility Across Nations 


The theoretical framework through which the researchers critically examined the gaps in accessibility 
across nations is based on Diffusion theory, most famously associated with the work of Everett Rogers (2010). This 
theory is commonly used to describe how discrepancies in the adoption of innovations occur (Lund, 2019). In the 
case of accessibility for educational systems, diffusion likely plays a major role, as the technologies used to support 
accessibility are generally free but information about the accessibility technologies and design are more restricted. 
Diffusion theory, and particularly supporting diffusion of information about accessibility, is suggested by the 
authors as one means to address the research problem identified above. 


Description of the Countries Studied 


The United States is the largest economic power in the world, with a Gross Domestic Product (GDP) of 
approximately $20.5 trillion US in 2018 (World Book, 2019). Its population is around 325 million (3" largest 
worldwide). It is a federal democratic republic, headed by an openly elected President and a Congress comprised on 
535 members, with 50 states that each have their own governor and congress to legislate matters not under the 
purview of the federal government. Virtually all inhabitants descend from non-natives of the land, with white (77%), 
black (13%), and Asian (6%) being the largest ethnic groups. Political and social strife and funding for education 
and healthcare are often seen as major concerns within the country. 

The People’s Republic of China is the world’s second largest economic power, with a GDP of 
approximately $14 trillion US (World Bank, 2019). GDP per capita, however, is much lower in China, as it has the 
world’s largest population at 1.5 billion. Officially, China is a one-party socialist republic, headed by the 
Communist Party of China. The federal government holds most of the power for governance of the nation. There are 
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over 50 officially-recognized ethnic groups in China, most all of which — including the largest ethnic group, Han, 
representing over 90% of China’s population — are native to the land. Some groups have emigrated from Japan, 
Vietnam, Korea, and India. A very small segment of the population originate from Europe, Australia, Africa, or the 
Americas. Suppression of certain political, religious, and social groups has, at times, been a concern within the 
country, as is the nation’s ability to adapt to its emerging role as an economic and higher education power. 

The Federal Republic of Nigeria is the world’s 31" largest economic power, with a GDP of almost $400 
billion US (World Bank, 2019). Currently, its population is just over 200 million, making it the 7" largest nation by 
population. After being a British colony for the first half of the 20" century and then run by military dictatorships 
for most of the second half, the nation finally achieved a stable democracy in 1999 and is now run as a federal 
presidential republic (not too dissimilar from the United States). Virtually all inhabitants of Nigeria are native to the 
land, although there are over 200 distinct ethnic groups recognized by the nation. Due to its status as a British 
colony in the early and mid-1900s, the official language of the country is English (though over 500 different native 
languages exist). Religion plays an important part of Nigerian culture, with the nation split virtually evenly between 
Christians (generally in the south) and Muslims (in the north). This has led to some social and political divide. 
Healthcare, access to information, and violence (particularly from terrorists groups like Boko Haram) are all major 
concerns for many within the country. 


Review of Prior Research on this Subject 


Jaeger (2012), Coleman and Berge (2018), and 3PlayMedia (2016) provide overviews of literature 
pertaining to accessibility, universal design for learning, and Web Content Accessibility Guidelines, focusing on 
Internet and learning resource use in the United States. While these guides are helpful for individuals in the United 
States who simply want an overview of these concepts, the significance for researchers in other countries is limited. 
A broader evaluation across several countries is likely to have greater significance for this group. 

Several studies have examined accessibility of university websites and learning management systems from 
around the globe, including Ahmi & Mohamad (2016) in Malaysia, Bhardwaj (2018) in India, and Kent, Ellis, and 
Giles (2018) in Australia. Each of these studies focuses only on a single nation, comparing the actual state of 
accessibility to an ideal. Comparative study, on the other hand, may give a more accurate assessment of what is 
realistic change, based on nations that are more/less advanced accessibility-wise. The assumption then becomes not 
that countries that are lagging in accessibility due to economic and social constraints should be held up to the ideal, 
but rather held to what other similar countries have accomplished (with the eventual goal still, of course, being the 
ideal). 

Other aspects of accessibility (physical spaces, access to libraries and other information centers) have been 
explored in a limited number of articles. Moorefield-Lang, Copeland, and Haynes (2016) described the process of 
creating accessible library instruction at the University of South Carolina. Bodaghi and Zainab (2013) studied 
accessibility to facilities in university library buildings in Iran. Simonson, Glick, and Nobe (2013) examined 
students’ perceptions of accessibility of buildings on a public university’s campus. Generally, the scope of these 
studies is limited, in some cases focusing only on a single university. This limited scope also limits the 
representativeness of the findings. 


How Data Was Collected 


The “data” for this study were government publications, peer-reviewed journal articles, monographs, 
newspaper articles, and grey literature relevant to addressing the research problem. Data was reviewed by all 
members of the research team. From the data, a complete narrative was developed that illustrates the strengths and 
weaknesses in the implementation of accessibility in educational systems. Based on knowledge of the social, 
political, economic, and information infrastructures in the three nations studied, diffusion theory was applied in the 
analysis to identify possible means through which government and education officials in each nation can support the 
adoption of strengthened accessibility policy and practice. 


Findings 
Accessibility Law 
United States 


Section 501. Section 501 of the 1973 Rehabilitation Act guides equal opportunity employment for 
individuals with disabilities for organizations/entities receiving federal funding. This section requires all 
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federal/federally-affiliated organizations/entities to have in place an affirmative action plan to ensure equal 
opportunity in the hiring, placement, and advancement of individuals with disabilities. Section 508 

Section 504. Section 504 is perhaps the most widely-known aspect of the Rehabilitation Act of 1973 and 
the most-commonly implemented aspect in schools and universities. This is otherwise known as the “reasonable 
accommodation” amendment (which would be further strengthened by the Americans with Disabilities Act). It 
directs all organizations/entities receiving federal funds to make reasonable accommodations, a modification in a job 
or work environment that will enable an otherwise qualified job applicant or employee with a disability to complete 
the application/interview process or perform essential job duties. 

Americans with Disabilities Act. The Americans with Disabilities Act of 1990 (ADA) was the most 
sweeping piece of disability civil rights legislation ever passed by the United States Congress. The legislation 
consists of 49 sections within five titles that cover employment, public services, public accommodations and 
services operated by private entities, telecommunications, and miscellaneous provisions 

Individuals with Disabilities Education Act. The Individuals with Disabilities Education Act (IDEA) was 
originally passed in 1975 as the Education for All Handicapped Children Act, and was reauthorized as IDEA in 
1990, the same year that the ADA was signed into law. IDEA’s passage and subsequent revisions significantly 
broadened opportunity for students with disabilities. The aim of IDEA is to provide equitable educational 
opportunity for students with and without disability by building supports into the educational process. 

W3C WCAG. The World Wide Web Consortium’s (W3C) Web Content Accessibility Guidelines 
(WCAG) were developed to guide accessibility compliance on the Internet. The accessibility guidelines are not in 
themselves legally enforceable, but have been adopted by many countries and organizations to guide their 
accessibility. There are 61 compliance criteria, sorted into twelve guidelines, which themselves are sorted into four 
principles. They are an important part of Section 508. 

Section 508. Section 508 is an amendment to the Rehabilitation Act of 1973 (the second of the two acts 
passed in that year). The original Rehabilitation Act was developed pre-Internet, pre-cellphones, pre-personal 
computer so the law was amended in 1998 to meet new needs with these emerging technologies, and again in 2017 
to better meet the technology requirements of the early 21* century. Section 508 requires federal agencies, and any 
institutions receiving federal funding, to provide equal access to all Information and Communication Technology 
(ICT) to individuals with disabilities. It suggests WCAG 2.0 Level AA compliance for providing this level of equal 
access. 


China 

People with Disabilities Act of the Peoples Republic of China. The Law of the People’s Republic of 
China on the Protection of the Disabled was amended and implemented in 2008 in accordance with the constitution. 
The law aims to protect the lawful rights and interests of disabled persons and ensure their equal participation in 
social life. It includes nine chapters and covers education, employment, social security, accessibility, and public 
entertainment. 

Regulations of Education of Persons with Disability. The regulations of Education of Persons with 
Disability were originally passed in 1994, and revised in 2011 and 2017. The regulations aim to guarantee the 
individuals with disabilities have equal opportunities to access to education with individuals without disabilities. It 
stipulates preschool education, primary education, secondary education, higher education, and vocational education 
for the disabled. 

Regulations on the Construction of Accessibilities. This regulation was passed in 2012 and aims to 
guarantee the accessibility of individuals with disabilities in social life. The regulation covers accessible 
infrastructure construction, information accessibility, and community service accessibility. In the part of information 
accessibility, public libraries shall establish reading rooms for the visual impaired individual, and provide braille and 
audio books; websites of disability organizations, governments, and public welfare shall meet the standards of 
website accessibility. 

Nigeria 

Nigerians with Disability Decree 1993. This Decree provides a clear and comprehensive legal protection 
and security for Nigerians with disability as well as establishes standards for enforcement of the rights and privileges 
guaranteed under this decree and other laws applicable to the disabled in the Federal Republic of Nigeria. The total 
14 sections cover the rights of individuals with disabilities in education, employment, transportation, social life, and 
communications. 

Discrimination Against Persons with Disabilities (Prohibition) Act 2018. This Act provides for the full 
integration of persons with disabilities into society and establishes the National Commission for Persons with 
Disabilities and vests the Commission with the responsibilities for their education, health care, social, economic and 
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civil rights. The Act covers the accessibility to physical structure, transportation, education, health care, and 
employment. 


Physical Accessibility to Places of Education and Information 


United States 
Approximately 17% of young adults (age 18-44) in the United States have some type of disability, including 10% of 
young adults with a hearing, vision, or mobility disability that might impair the ability to participate in non- 
accessible higher education (Okoro, Hollis, Cyrus, & Griffin-Blake, 2018). Individuals who identify as White or 
Asian have the lowest incidences of disability, while individuals who identify as Black, Hispanic, Native American, 
or Multiracial, have an average incidence that exceeds 20%. Disability is yet another barrier that increases racial 
disparity (Fredrick & Shifrer, 2018). 

It was within the United States that the predecessor of Universal Design principles was developed, by Timothy 
Nugent and colleagues at the University of Illinois (Reagan, 2017). Universal Design for Learning, the principles 
that guide accessibility design of instruction (especially in online learning settings), was also developed in the 
United States (Meyer, Rose, & Gordon, 2018). It was also a leader in the passage of disability rights and 
accessibility legislation, such as the 1973 Rehabilitation Act and 1990’s Americans with Disabilities and Individuals 
with Disabilities Education Acts. This, however, does not mean that the United States is necessarily perfect in its 
adherence to accessibility guidelines. Particularly among older construction, accessibility can be a major barrier 
within the United States. 

Universities tend to be some of the oldest buildings in the United States, with many constructed before 
accessibility measures like elevators and sloping entranceways (ramps) were invented or financially practical 
(Fisher, 2019). A disparity exists among more and less financially successful universities, where those universities 
fortunate to have a high level of financial success or receive sufficient funding from state sources or grants were able 
to modernize old buildings with accessibility elements. Less financially successful universities may be stuck with 
buildings that limit accessibility. Lawsuits over physical accessibility to university buildings are not uncommon. 
Nonetheless, U.S. universities excel in physical accessibility compared to nations where buildings have stood for 
many centuries, as is the case in many European and Asian nations. 


China 

By the end of 2010, there were over 85 million individuals with disabilities in China, including 12.63 
million have a visual disability, 20.54 million have hearing disability, 1.3 million have a speech disability, 24.72 
million have a physical disability, and 26 million have other disabilities (China Disabled Person’s Federation, 2012). 
These individuals account for 6% of the total population of China. This incidence rate is relatively consistent with 
the percentage of Asian-Americans with disability. 

A large number of physical accessibilities, such as ramps, elevators, tactile pavements, and restrooms for 
disabled individuals, was built between 2008 and 2010 for the Paralympics (Blauwet & Willick, 2012). However, 
several problems can be observed in those established ones, such as lacking standard ramps and braille for public 
facilities, design deficiencies and artificial obstacles of tactile pavement, and inaccessibility of public restrooms. The 
occasions frequently occur for a wheelchair cannot access to a ramp of a supermarket, for instance, as the 
supermarket sets up a circle of barricades outside the door to prevent the loss of shopping carts, and the slope is too 
steep to access. Most high-rise buildings install elevators for both individuals with and without disabilities, but some 
of them do not provide braille on the buttons, which stops persons with visual impairments accessing to it. Another 
occasion may occur that the braille on a public facility is inaccurate, such as rotating the braille 90 degrees, using 
English spelling for Chinese, and braille points out of place. 

Tactile sidewalks are built in most walkways for person without disabilities, however, many of them laid 
discontinuously, which causes difficulties for visually impaired individuals to follow; some of them do not connect 
to a public facility, such as a hospital; and some of them are illegally occupied by either public facilities or private 
facilities. Many public facilities set accessible restrooms for individuals with physical disabilities, but part of them 
are held unlawfully and causes inaccessibility to disabled persons; some of the public bathrooms have signs for 
accessibility sign outside but do not install accessible devices, such as handles, inside. 

The state of the economy is the leading cause of these deficiencies. Even though China has become the 
world’s second-largest economic power, Chinese people were facing dearth before the 1970s. A government cannot 
consider the higher-level needs, such as accessibilities for disabilities, when it is struggling in feeding its people. 
Comparing with to the over 200-year history of the U.S. Disability Laws/Regulation/Act (Meldon, 2017), 50 years 
(start from the 1970s) is not enough for China to form a relatively complete legal system to protect disability’s rights 
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and public awareness of the accessibility. The Chinese government has not stopped promoting to improve public 

awareness and investing heavily in protecting disabled person rights and improving circumstances. By the end of 
2018, 2,364 comprehensive service facilities, 914 rehabilitation facilities, and 791 nursing homes for the disabled 
had accomplished with a total investment of 5 billion U.S. dollars (China Disabled Person’s Federation, 2019). 

Nigeria 

Recent studies have shown that the population of people with disability in Nigeria grows at a steady rate of 
1.08% every year since 2011 (Smith, 2011; Books2A frica, 2013; National Population Commission of Nigeria, 
2018), currently amounting to 19 million or 9.5% of the 200 million Nigerians who are living with disabilities across 
six geopolitical zones (National Population Commission of Nigeria, 2018). Therefore, people with disability in 
Nigeria forms a large population of the less privileged- those who are less likely to access educational and 
informational resources and centers (Smith, 2011). 

Prior to the enactment of the 1993 Disability Decree which empowers persons with disability with their 
human rights, including right to access places of education and information center and information resources, people 
with disability in Nigeria were mostly left to the mercy of their family or anyone who could provide them help to 
access public facilities such as information centers, health, and educational buildings (Rwomire & Raditlhokwa, 
1996). However, the Discrimination Against Persons with Disabilities (Prohibition) Act (Federal Republic of 
Nigeria, 2018), among many things enforces the need for public facilities- education and information places own by 
the government, and any private body that render services to the public provide the necessary accessibility structure 
for people with disability. Facilities such as ramps, wheelchairs, parking spot for disabled persons, lifts for people 
with disability, braille signages and signs for the visual impaired are encouraged to be part of public places, 
especially within and around educational buildings and information centers. While such is encouraged by the federal 
government, however, not all public information and education centers provide facilities or renders services that 
would encourage accessibility by persons with disability in Nigeria (Ijadunola, Akintan, Afolayan, & Akanji, 2019). 


Access to Learning Management Systems 


United States 

Canvas. Canvas LMS is designed to conform to WCAG 2.0 and Section 508 guidelines. Website provides 
links to compatible screen readers and walkthroughs of the features instructors can use to create accessible content. 
https://community.canvaslms.com/docs/DOC-2061 

Blackboard. All Blackboard products are designed to conform to WCAG 2.0 Level AA and Section 508 
standards; regular audits are conducted by a third-party organization. The website also provides information for 
educators about creating accessible course content. https://www.blackboard.com/accessibility.html 

Moodle. Website provides some information about accessibility, but it is not easy for a user to understand. 
Making sure the LMS conforms to accessibility guidelines is a work in-progress. Some links to screen readers are 
provided for users. 


China 

There are few comprehensive learning management systems applied in higher education institutions, 
similar to the Blackboard and the Canvas, found. More systems are platforms provide specific professional training, 
such as English training, or coding training, which are similar to a systematic tutorial videos on YouTube. This 
situation may be caused by insufficient technological and educational resources. 


Nigeria 

Due to limitations in economy, not many learning management systems in higher education institutions are 
found in Nigeria. Most of them relate to corporation training, self-taught learning, or mobile learning as a 
supplement of primary and secondary education. In instances where LMS are used, it is generally Moodle, as this is 
an open-access, open-source LMS. Accessibility of Moodle is discussed in the U.S. section above. 


Access to Library Management Systems 
United States 


Ex Libris: Ex Libris Library Management System conforms to WCAG 2.0 and Section 508 standards with 
exceptions. These exceptions are not explicitly mentioned. 
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SirsiDynix: SirsiDynix does not specifically mention if and which guidelines the LMS follows. Instead, the 
websites indicates that if you experience a problem, you should contact the webmaster; if certain information is not 
accessible, customer service can provide it in an accessible format. 

Innovative Interfaces: Innovative Interfaces admits that it does not fully conform to WCAG 2.0 or Section 
508, but is constantly working to make changes to comply. 

OCLC: OCLC states that it works to continuously adhere to WCAG 2.0 and Section 508 guidelines. Also 
provides Voluntary Product Accessibility Templates (VPAT) for each product. 

Koha: Koha is an open-source library management system shell. As such, the accessibility of the system 
relies on the design given by system managers at the library. 


China 

CALIS (China Academic Library and Information System): It is aiming to promote and improve resource 
sharing among academic libraries, reduce expenses for participating libraries, and support development of higher 
education in China. It intends to build an infrastructure for resource-sharing and see multiple resource-sharing 
functions among participating libraries. It does not specifically mention if and which accessibility guidelines 
follows. 

CASHL (China Academic Humanities and Social Sciences Library): It aims to cooperatively acquire, 
preserve and share foreign and Chinese periodical resources in the humanities and social sciences among member 
libraries, and provide a unified online portal for users to retrieve and utilize these resources. It does not specifically 
mention if and which guidelines of accessibility follows. 

ILAS II (Integrated Library Automation System): The website introduces the technology applied in ILAS 
II and its functions, but does not specifically mention if any accessibility policies or regulations follow. 

InterLib: The website introduces the developing computer languages, operation platform, system structure, 
etc., but does not mention if any accessibility policies or regulations follows 


Nigeria 

KOHA. The module of Frontoffice promotes the inclusion of all citizens. It is developed in accordance 
with the WCAG 2.0 

Visionary Technology in Library Solution (VTLS). It does not specifically mention if any accessibility 
policies or regulations follow. 

Millennium. It does not specifically mention if any accessibility policies or regulations are followed. 


Access to Educational Websites 


United States 

Most university websites in the United States do a poor job of adhering to accessibility guidelines, like the 
WCAG 2.0. Lawsuits by disability rights groups towards universities is very common, including over 50 lawsuits 
filed by Jason Camacho in December 2018 against major universities such as Cornell and Vanderbilt University 
(McKenzie, 2018). In some cases, accessibility to websites in the United States can be worse than many other 
nations, because these websites incorporate dynamic elements (e.g., javascript) that have more limited compatibility 
with screen readers and other accessibility tools. 

Educational websites not produced by universities, like Chegg and Google Scholar, are not necessarily held 
to the same accessibility standards. This, however, does not necessarily mean that accessibility is worse for these 
websites. As discussed, university websites’ accessibility can be poor due to over-complexity, while Google’s 
simple interface (a header and a search box) actually lends well to accessibility. 


China 
A university’s website is generally providing the brief introduction about the university (history, current university 
leadership, successive leadership, and organizations), schools and departments, faculties, scientific researches, 
university news, enrollment, and job opportunities. The websites are more often used for promoting, which allow 
people to know more specifically about the university, such as new research subjects and on/off-campus activities. 

Most designs of universities’ websites are following the Universal Design principles, presenting various 
levels headings according to the contents and an aria-labels/labelledby for users of screen reader and keyboard, but 
most alternative texts for an image or an image link are missing. This situation may be caused by lacking relevant 
regulations and handbooks for website accessibility and social awareness. 
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Nigeria 

Education websites owned by the various educational institutions in Nigeria provide little or no 
accessibility aid for people with disability. Some of the universities and other academic institution, and even 
information centers in Nigeria don’t have an official website with a recognized domain name as encouraged by the 
Nigerian federal ministry of education. Therefore, most educational websites in Nigeria are not complaint to the 
accessibility needs of persons with disability, nor are they up to the recommended standard to be consider as an 
educational website. 


Discussion 


Comparing and Contrasting the Countries 

Across the three countries of the United States, China, and Nigeria, financial situation appears to play a role 
in the capacity to provide accessibility. When a nation’s universities can hardly afford a learning or library 
management system in the first place, there is limited emphasis on accessibility to these systems. Among United 
States institutions of higher education, those with lower financial standing tend to have more barriers to 
accessibility. In general, it appears that most institutions want or feel compelled to conform to accessibility 
guidelines, but it may not be financially feasible. 

Among learning and library management systems, most do fairly well at complying with accessibility 
guidelines, or are in the process of doing so. The greatest challenge may be among the open-source systems, as the 
front-end design is left to each institution to design, meaning that accessibility can vary based on the quality of the 
design. These systems, however, are immensely valuable in developing countries like Nigeria, where educational 
institutions may otherwise not be able to afford any sort of system. Expansion in the use of these open-source 
educational systems is likely to only increase in nations like Nigeria, so local knowledge of accessibility principles 
at individual universities could prove vital to ensure the front-end design is accessible. 

Institutions in the United States likely benefit from being located in the nation in which the major 
accessibility guidelines were developed. Knowledge of these guidelines appears limited in the other countries 
studied, with many educators appearing to learn the principles while studying abroad or online (Rao, Currie-Rubin, 
& Logli, 2016). This indicates that there may be limits to the diffusion of accessibility concepts to countries outside 
the United States. 

How to Improve the Diffusion of Accessibility Concepts 

The rate of diffusion of information in developing countries may be influenced by increasing access to 
information and educating information gatekeepers (professors, librarians, government officials) (Lund, 2019). As 
such, advocates for accessibility may benefit from the targeted dissemination of accessibility information to 
educators and institutions of higher education that do not presently observe them. As noted by Nishita, Liebig, 
Pynoos, Perelman, and Spegal (2007), opposition to accessibility policy can hamper diffusion; thus, it is important 
not only to educate the educators about accessibility, but also to be prepared to address the concerns and critiques of 
those who oppose accessibility. In regard to accessibility to educational systems, the greatest concern may be that 
accessible design takes valuable time away from the creation of content to help a small group of students. As found 
by Lazar, Dudley-Sponaugle, and Greenidge (2004), many web developers indicate that they would only make a 
website more accessible if a law compelled them to, because it is additional work and they believe the time is not 
worth the benefit. Advocates can push back against such critiques by pointing out, as noted by Lisney, Bowen, 
Hearn, and Zedda (2013), that accessible design produces positive outcomes for all users, not just those with 
disabilities, by forcing designers to engage in more purposeful and inclusive design. 

From this comparison of accessibility in the United States, Nigeria, and China, advocates may be able to 
cite specific gaps in accessible design in their nation, and use examples from the other nations to identify what 
should or should not be done to improve these gaps. Each nation has clear gaps and can learn from the others. As 
Everett Rogers himself argued, diffusion of innovations is a theory of information exchange (Rogers, 2010). By 
exchanging information about accessibility principles and disparities among nations, the outcome will hopefully be 
greater adoption of these principles. 


Conclusion 
Discrepancies in conformance of educational systems to accessibility guidelines exists both within and 
across nations. The United States has the greatest conformance across education systems, but physical and website 


accessibility can vary based on a university’s financial standing and the number of dynamic web design elements 
incorporated into a website. China and Nigeria both have limited conformance to accessibility guidelines, but are 
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also limited financially and in knowledge of accessibility principles. Greater collaboration among institutions 
worldwide is needed to facilitate the diffusion of information about accessibility principles and provide support of 
implementing these principles. Governments in these nations have a financial obligation to support improvements to 
physical accessibility. By implementing common-sense changes, conditions can be improved for learners with 
disabilities as well as those without disabilities. 
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Abstract 


This paper explores the intersection between current methods for introducing and teaching computational 
thinking (CT) to students in kindergarten to twelve grade (K-12) and culture. Available literature in CT, cognitive 
theory, and K-12 pedagogy will be used to gather and analyze the methods available today. Our goal for this paper is 
to gain an understanding of the current state of best practices for introducing CT to propose the best framework for 
scholars and practitioners to focus on. 


Introduction 


Initially introduced by Wing (2006), computational thinking (CT) is a set of thinking skills and approaches 
that are deemed necessary to solve complex problems using a computer. Wing (2006) proposed that these skills 
would be valuable for an individual’s chance of employment, as society will demand more usage of computers to 
solve its needs and issues. Despite the groundwork laid, the efforts to introduce and teach CT to students have faced 
considerable challenges. One that is crucial is “What are effective ways of learning (teaching) computational 
thinking by (to) children” (Wing, 2008, p. 3720). Scholars have stepped up to the challenge and proposed different 
methods in answering this particular question. Lee et al. (2011) showed that introducing CT to youths in practice 
starts with modeling and simulation, robotics, and game design and development. Lu and Fletcher (2009) called for 
a different approach in introducing concepts of coding by not using programing language and opt to use everyday 
language that is appropriate to the grade level. Weintrop et al. (2016) argued that CT is a valuable asset that is 
beneficial enough to be presented by itself and also to augment mathematics and science classes. 

This paper explores available methods in introducing CT. The research question that drives this study is, 
“what are the common themes amongst the available approaches in introducing CT to K-12 students?” This study 
will then assess these approaches along with skills associated with CT, such as abstraction and decomposition, 
heuristic reasoning, and knowledge of computer science concepts, and synthesize the findings with cognitive 
theories and K-12 pedagogy literature (Weintrop et al., 2016). Aspects associated with the skills that can be 
analyzed are the close connection with programing language, the necessity to attain and retain concepts, and the 
usage of CT in problem-solving. 

As CT is strongly connected with coding and programming language, in addition to a large number of 
methods of introducing CT involves coding, it is necessary to explore possible ways and models in teaching a new 
language to students. One model that has been used extensively in teaching English for students with English as a 
native language and students with English as a second language is the Sheltered Instruction Observation Protocol 
(SIOP) Model (Echevarria, Vogt, & Short, 2008). Supported by fifteen years of research, SIOP model articulates the 
necessary components in not just furthering knowledge of English for native English speakers but also introducing 
and retaining a foreign language for non-native students (Echevarria et al., 2008). The scientific nature of this model 
is encouraging as it can help the effort in creating a working model for introducing CT to K-12 students. 

In addition to programming language, CT also involves knowledge and specific information that students 
will need to attain. The idea of achieving new knowledge and issues that can arise from it is not wholly unique in the 
world of education. One model that tries to address the attainment of concept is the concept attainment model. The 
aforementioned model seeks to address the attainment of concept through three phases: presentation of data and 
identification of concept, testing attainment of concept, and analysis of thinking strategies (Joyce & Weil, 1980). 

The process of transferring CT does not stop after students have attained the knowledge and skills. As 
argued by Wing (2006), CT skills are used for numerous purposes, namely problem-solving. In facilitating transfer 
and problem-solving in the classroom, Ormrod and Davis (2004) proposed several suggestions. Some of these 


413 


suggestions are students need to learn information meaningfully and thoroughly, students should also learn problem- 
solving strategies at a meaningful level, students have a mental set for transfer, and some prerequisites skills should 
be practiced until they are learned to the point of automaticity (Ormrod & Davis, 2004). 

Looking at commonality amongst the available methods in introducing CT and analyses of aspects of CT 
through the lens of cognitive theory and K-12 pedagogy that available methods try to address, this paper will then 
propose the best framework for practitioners and scholars to use in the classroom and future research in CT. 


Method 


This paper uses a literature review approach to gather the data by analyzing the methods and findings of the 
available literature (Lye & Koh, 2014). Our goal is to unearth the ways that are used in teaching CT across cultures 
and to determine whether or not culture is considered when it comes to this emerging field. To be included, papers 
must have been peer-reviewed and be more than two pages, to eliminate editorials, book reviews, and viewpoints 
(Aloini, Dulmin, & Mininno, 2007). This paper gathers publications from the period of 2015 to 2019 to get the latest 
trend in CT, rather than focusing on the formative years. On cognitive theory and K-12 pedagogy, this paper utilizes 
both peer-reviewed papers and notable books that might have preceded the establishment of journals in their 
respective fields. In total, 49 peer-reviewed articles on teaching and introducing CT were selected. Although the 
author was able to find more than the selected number of articles, some were removed because they were written in 
foreign languages in which the authors are not proficient. Also, some were excluded because they discussed CT as a 
tool to assess other variables rather than the focus of the study. 


Findings 


From the findings, it is apparent that hands-on delivery is the preferred method of teaching and introducing 
CT. Despite the fact, having diversity in the method of delivery is needed to bring a more balanced and complete 
approach. The only lecture-based transfer of knowledge studied was on pre-service teachers to better equip them 
with CT knowledge and skills (Yadav et al., 2014). Perhaps, the 17 cases of both lecture and hands-on activities are 
an encouraging sign that lecture still has its place in CT knowledge transfer. 
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Figure 12. Framework for introducing and teaching CT 


The framework that this study proposes highlights the importance of both types of delivery. Recognizing 
that no two students are alike in how they acquire, process, and retain knowledge, the framework will offer a holistic 
mean of introducing and teaching CT, as shown in figure 12. Similar to SIOP, teachers will use CT concepts and 
building background to create comprehensible inputs. This way, the concepts of CT will only be delivered according 
to the student's level, cultural background, and language proficiency. Once the inputs have been compiled, the 
teachers can then deliver the lesson. Delivering the lesson will start with a lecture, followed by a hands-on activity. 
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The hands-on activity is designed as a scaffolding to help students learning the concepts and skills of CT in an 
environment that nurture their self-efficacy. The idea of using scaffolding is not entirely new in CT as some of the 
selected articles for this study used it in their research (Bagley & Rabin, 2016; Basu et al., 2016). 

The scaffolding itself is a loop. After the teacher demonstrated how to apply a concept of CT, the students 
will then use the concept under the guidance of the teacher. One the teacher is satisfied with the students' 
performance, the students will then apply the CT concept independently. The loop will only break when the teacher 
is confident with the performance of the students. In a case where an additional lecture needed, the teacher has an 
opportunity to do so. The cycle will then repeat after a concept has been thoroughly processed by the students. 


Building Background 

When building the background to be used to create comprehensible inputs, teachers will need to take 
account of the student's level, cultural background, and language proficiency. The student's level will determine 
which concepts to be introduced and taught. Basic concepts, such as "problem-solving" and "decomposition," are 
appropriate for students in lower grades. More complex concepts, such as "reusable functions" and "objects," may 
not be suitable for students in lower grades and can be introduced later in K-12. Using terms and concepts of CT by 
Selby and Woollard (2013), an example of how to break them by levels is shown in table 1. 


Table 1. An example of terms and concepts by level, culture, and language proficiency 


Terms/Concepts Grade level | Culture Adjusted | Proficiency Adjusted 
Country A AP students 

A thought process 2-6 2-6 K-6 
Abstraction 2-8 2-8 K-8 
Decomposition 2-8 2-8 K-8 
Logical thinking K-6 K-6 K-6 
Algorithmic thinking 6-12 4-12 4-12 
Problem-solving 4-12 K-12 2-12 
Evaluation 4-12 K-12 2-12 
Generalization K-6 K-6 K-6 
Systems design 9-12 6-12 6-12 
Automation 8-12 6-12 6-12 
Computer science content 8-12 6-12 6-12 
Modeling and simulation 9-12 9-12 9-12 


Once teachers have broken up these concepts by level, they will then have a holistic picture of the students’ 
progression from K to 12 grade. This will then be used in the next factor they will need to consider, cultural 
background. Some ideas and concepts can be more natural in one culture when compared to others. Teachers need to 
recognize this and adjust the level where the concepts should be introduced. Finally, the language proficiency of the 
students will need to be taken into account. Because of the complex terms and ideas that are inherent in CT, students 
who perform better than their peers can certainly grasp them sooner and better. After the teachers have taken all of 
the aforementioned into the model and adjusted the grade accordingly, the final version should look like table 1. 


Lesson Delivery and Hands-on Activities 

As discussed before, delivering the lesson will start with a lecture. Having the appropriate level of ideas 
and concepts for the level, culture, and language proficiency should keep the students' engagement between 90 to 
100 percent. This lecture will then be augmented by hands-on activities that are in line with the lecture. Teachers can 
include many activities in the lesson. Some activities that this study had presented, such as programming using a 
programing language, programming using software (scratch), robotic, and modeling software, can augment the 
delivery of ideas and concepts. While it is admirable to try and use new activities to cultivate CT skills, teachers will 
need to stick to what has been proven to work. An excellent example with less than desirable results is how Tsai et 
al. (2017) tried to introduce CT by using Microsoft office products such as PowerPoint. Once again, the teachers 
need to know different tools and activities that can help in teaching and introducing CT. In addition, the teachers 
themselves will need to be comfortable using the tools and doing the activities. The hands-on activities are 
considered to be scaffolding activities that will slowly shift the responsibility to the students. From the review of 
literature in CT, scaffolding is something that is used by several selected studies. Scaffolding itself is something that 
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SIOP model suggests teachers use in their classrooms. In the proposed framework, the hands-on activity portion is a 
loop that can only be broken when the teacher is satisfied with how the students are progressing. Review and 
assessment is next after the hands-on activity. 


Review and assessment 

Much like any other subject, review, and assessment of the delivered material are needed to ensure proper 
delivery. Teachers will need to set up baselines and monitor the progress of the students. This information, in 
addition to the changing grade level, culture, and language proficiency, can be used to modify the terms and 
concepts break down. However, the primary purpose is to signal the continuation of the delivery of knowledge and 
repeat the process. 


Discussions and Conclusion 


After reviewing the literature associated with the teaching of CT, this study proposed the best framework to 
teach and introduce CT to K-12 students. This framework, combined both lecture and hands-on activities, aims to 
construct systematically comprehensible inputs that students can digest based on their level, cultural background, 
and language proficiency. Although this framework is aimed at the students, teachers can undoubtedly benefit from 
the framework also. The review of literature revealed that ensuring the teachers comfortable with the idea of CT and 
the delivery of CT could further help the cause of introducing CT to students, especially younger students. With that, 
this framework will only be effective when the teachers themselves are comfortable with the framework. 

Regardless of whether CT will stand on its own or be included in an already established subject such as 
math and physics, this framework allows the teacher to deliver CT knowledge effectively. Also, this framework will 
reiterate the point that CT is not programming. The element of culture, language, and concepts will hopefully drive 
home the point that CT is much more than just learning how to program. Although programming can be helpful as a 
way to introduce concepts of CT, it should not be the focal point of the learning process. When looking at it from the 
perspective of the proposed framework, programming should be contained in the loop portion of the scaffolding 
activity. 


Limitations and Future Direction 


The article included in this study were only those available to the authors through the university’s library 
research databases in the span of one semester. Also, as asserted before, only studies that were written in English 
were selected because of the limited language proficiency of the authors. In other words, studies written in other 
languages such as Chinese, Korean, Swahili, Spanish, and other languages did not make it into the final list. This 
study would like to highlight these studies and that the reality of them not making it into the final list does not 
diminish the contribution of the articles to the overall knowledge of CT. 

The goal of the proposed framework is to help in teaching and introducing CT to students of all levels. The 
next step in this study is to apply the framework in the classroom and analyze it for its effectiveness in conveying 
ideas and skills related to CT. The hope is that the next generation will be better equipped in solving not just 
individual issues, but also issues in the society where they reside. 
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According to the Power of International Education (2019), the number of international students in the US 
reached a new high of more than a million. These students studying alongside their American classmates are an 
asset and add to the diversity of the learning environment. Their choice of studying at an American university is 
reflective of the dedication of American colleges and universities to students’ academic, professional, and personal 
successes (IIE, 2019). This diverse group of students not only represent different race, gender, ethnic group, age, 
religion, sexual orientation, citizenship status, mental and physical attributes, but also their national and individual 
cultures. Diversity, however, is not representative only of cultures outside the US, but also of those from within. 

Intercultural communication is an important aspect of learning for these diverse groups of students. 
Hofstede (1991) defined culture as that collective programming or behaviors of a group of people that distinguishes 
them from others, and according to CARLA (2019), Center for Advanced Research on Language Acquisition, 
culture is the shared patterns of behaviors and interactions, cognitive constructs, and affective understanding that are 
learned through a process of socialization. These shared patterns identify the members of a culture group and 
distinguish them from another group. Within these cultural frameworks are patterns of behavior for subgroups 
(gender, age, religious practice, socio-economic status etc.) which argue that not only are there inter-cultural 
variables, but there are also intra-cultural variables. Learning and related communication skills are part of this 
framework of behavior. Despite their observable presence within educational institutions, diverse students are often 
outliers, even when their numbers are large. They are not necessarily “needy” students, novice or remedial learners; 
rather, they are often academically strong and hold expert knowledge. 

Diverse students are part of both face to face and online classes at most educational institutions. According 
to Dusst & Winthrop (2019) online education has become an increasingly accepted option, especially when 
coursework can be “stackable” into degrees (Friedman, 2016; Ruff, 2016). Online classrooms provide opportunities 
for a confluence of learners from both national and international contexts. This confluence produces rich learning 
experiences and provides benefits by opening up physical and temporal boundaries to provide access to learning for 
large numbers of students. As with face to face classrooms, communication remains an important part of these 
students’ learning experiences. Additionally, learning is largely effected by cultural norms of the instructors and the 
students. Online classrooms are not unlike face to face classrooms and diverse learners, where according to Vatrapu 
& Suthers (2007), they feel the profound effect of culture on technological learning environments because of 
cultural attributes, such as interaction patterns, meaning construction, classroom interaction patterns, and other 
social-cognitive factors that influence their learning. In the online classroom, the question is about how do we 
bridge intercultural communication to facilitate learning, particularly when students are not connected in space and 
time. In online synchronous learning, the variables related to communicating with instructors and peers remain 
somewhat similar to those in face to face learning. However, in online asynchronous learning, where contact with 
the instructor and peers are minimized and where learners learn from instructional modules and materials created by 
instructional designers, there is a need to consider variables that effect intercultural learning. Factors, such as 
bridging communication gaps between learners, between the learner and instructional materials, as well as between 
the learner and the instructor (Vatrapu & Suthers, 2007, Milheim, 2014) need to be addressed and incorporated 
during the design and development of online courses. According to Borden (2016) innovative solutions are needed 
to create a culture of “best” learning by embracing pioneering practices grounded in neuroscience, learning research, 
and best practices in education technology. 
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In line with the national trend, a large influx of international students enrolled in undergraduate and 
graduate programs at a small private institution, for which this was relatively a new charge. A large number of these 
students are enrolled in online classes as part of their program of study. These online classes are developed with a 
partnered effort between a faculty and an instructional designer. The course design process involves the faculty as 
the subject matter expert, who provides the course content and the instructional designer, who as the learning theory 
specialist. They work cooperatively to design and develop online courses using the most appropriate framework for 
the learning solution. For faculty and instructional designers, who are not trained in culturally sensitive pedagogy, it 
is not uncommon as professionals to assume that the online learning environment is the same for everyone, but how 
about the users in the environment — do they perceive something different? (Vatrapu & Suthers, 2007). 

For the purpose of the training at the institution described above a two-pronged approach was employed 
using Hofstede’s cultural dimensions (Table 1) and an inclusive learning model, a framework from education that 
looks at the classroom as a whole. The premise for the training was based on the notion that learning is governed by 
culture; therefore, learning environments are not culturally neutral, that computers and other digital devices are 
cultural tools, and that instructional designers are not culturally neutral individuals. The reality of the online 
classroom is inclusive, in which culturally diverse students share the same learning context as mainstream American 
students. The goal of the workshop was to train instructional designers to use culturally sensitive learning pedagogy 
for all learners, including diverse learners from international contexts by addressing two key areas: culture and 
knowledge acquisition. 

Dimension 1: The Culture Domain 


Hofstede (1986) discussed interaction differences between the teacher and students and related them to his 
original 4-D model of cultural differences among societies. According to Hofstede, these differences between 
teacher and students emanates from four dimensions of cultural differences: Individualism versus Collectivism, 
large versus small Power Distance, strong versus weak Uncertainty Avoidance, and Masculinity versus Femininity. 
These four dimensions were used in the development of the workshop for instructional designers. 


Table 1. Hofstede’s Cross-cultural communication model 


Hofstede’s Dimensions Explanations of the Dimensions 


Individualism/Collectivism This dimension is about the extent to which people feel 
independent, as opposed to being interdependent as 
members of the larger group, where members 
recognize and respect their space within the group. 


High power/Low power distance This dimension is the extent to which more powerful 
members of society within organizations and 
institutions provide leadership to the less powerful 
members, who accept and expect that power is 
distributed unequally. 


High certainty/low certainty This dimension is the extent to which there is low 
tolerance for the unknown, which causes anxiety and 
distrust, with a wish to have fixed routines and rituals. 


Masculinity/Femininity This dimension is the extent to which cultures with 
high masculinity are driven by competition, 
achievement and success, with success being defined 
by the “winner” or “best-in-the-field.” 


Adapted from https://geerthofstede.com/culture-geert-hofstede-gert-jan-hofstede/6d-model-of-national-culture/ 


Culture Domain1: Individualism vs collectivism 


Interventions for addressing cultural traits 
Learning traits for learners from What can subject matter What can the instructional designer 
collectivist cultures functioning in | experts/course faculty do when do during the design and 
individualist cultures. organizing content for the development phase of the course 
instructional designer? with the content? 
Group work works well Build opportunities for project- Work with faculty to 
based learning (PBL) or group adapt course features with features 
projects with course content. of PBL using project management 
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tools in the course or assignments or 
modules. 


Conformation from groups is 
important 


Relationships with in-group 
members are intensive and 
interdependence is high 


Establish group roles for students to 
reduce confusion with conception 
and initiation, planning, execution, 
as well as monitoring the project. 


Establish clear expectations and 
guidelines for group roles and 
opportunities for peer 
feedback/evaluation systems 


Judicious choice of e-learning 
framework. For example, in a 
gamified course design provide a 
balance of cooperative opportunities 


and competitive experiences. 


Culture Domain 2: High Power and 


Low Power Distance 


Interventions for addressing cultural traits 


Learning traits for learners from 
high power distance functioning 
in low power distance cultures 


What can subject matter 
experts/course faculty do when 
organizing content for the 
instructional designer? 


What can the instructional designer 
do during the design and 
development phase of the course 
with the content? 


Effective learning is related to 
expert knowledge, such as 
dependence on established 
knowledge from textbooks, 
information provided by subject 
matter experts, such as direct 
expert lectures. 


Balance teacher-centered approach, 
by providing information related to 
content through video or audio 
lectures with independent learning 
opportunities with instructional 
materials. 


As learning experts, balance 
different viewpoints of learning: 
balance direct teaching with 
opportunities for independent 
learning. 

Understand the learning gap being 
bridged and derive appropriate 
learning outcomes. 

Implement instructional strategies 
that balance direct teaching of 
declarative knowledge with 
experiential learning associated 
with procedural or problem-solving 
types of learning. 


Sharing opinions, thoughts or 
ideas in groups or with instructor 
in the large group. 


In asynchronous settings, provide 
opportunities to set up online 
meeting times to work with student 
questions after class. 

In synchronous settings use the 
meeting time to work with student 
questions. 


Design opportunities within the 
course to ask questions at point of 
need through communication tools 
embedded in the LMS or build 
metacognitive checks (ex. “What 
was most confusing to me about the 
material explored in the module?”’) 


Students hesitate to participate in 
group work or group discussion 


Lower affect associated with group 
work by providing explicit 
directions about group participation 
and assisting with group roles. 


Embed directions related to explicit 
instructions related to group work. 


Individual learner creativity and 
independent learning 


Provide “structured” interaction 
opportunities through carefully 
crafted pedagogical use of 
communication tools, such as 
discussion board activities, or chats. 


Use resources within the Learning 
Management System to design 
learning opportunities to leverage 
communication related to 
instructional materials. 


Learning traits for those from 
high uncertainty avoidance 
function in low uncertainty 
avoidance cultures 


Culture Domain 3: High Certainty Avoidance vs. low Certainty Avoidance 


Interventions for addressing cultural traits 


What can subject matter 
experts/course faculty do when 
organizing content for the 
instructional designer? 


What can the instructional designer 
do during the design and 
development phase of the course 
with the content? 
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Learners look for preciseness in 
instruction and course related 
activities 


When preparing content module 
pages, simplify and clarify 
language to modify the rigor of 
instructions, explain academic 
vocabulary or other content related 
jargon. 


Determine where the difficulties lie 
in the way the instructor has 
provided information and apply 
visual thinking design principles, 
but don’t water down rigor. 

Using e-learning support with 
content related vocabulary, provide 
in demand micro learning units. 


Learners have less tolerance for 
ambiguity and low risk-taking to 
avoid failure. 


Provide explicit directions with 
assignments and other tasks. 


Use resources within the LMS to 
design learning opportunities to 
leverage communication related to 
instructional materials. 


Learners believe that teacher 
knows everything. 

Learners find it difficult when the 
teacher says “I don’t know.” or “I 
am not sure.” 


Instructor needs to be the “sage” 
and provide responses since the 
student looks up to the teacher. 


Instructional designers need to 
design balancing direct instruction 
and interaction from the “expert” 
source. 


Student accuracy is rewarded. 
Strong need for affirmation and 
consensus 


Design and build reward systems. 


Culture Domain 4: Masculinity vs Femininity 


Interventions for addressing cultural traits 


Learning traits for those from 
cultures with feminine 
orientation function in masculine 
orientation cultures 


What can subject matter 
experts/course faculty do when 
organizing content for the 
instructional designer? 


What can the instructional designer 
do during the design and 
development phase of the course 
with the content? 


Learners do not prefer to stand out 
from the crowd therefore 
cooperation rather than competition 
is preferred 


Learners have a great deal of 
overlap between male and female 
roles and place importance on 
relationships with those they work 
with. 


Organize learning activities to 
respect cooperation and 
collaboration. 


“groups” 


Be involved with project groups 
and provide help with structuring 
conversation, as needed. 


Judicious choice of e-learning 
framework. For example, in 
gamified course design provide a 
balance of cooperative 
opportunities (team up) and 
competitive opportunities. Work 
with the instructor to create 
opportunities to compete in 


Use of cooperative learning 
strategies through project-based 
learning. 


Dimension 2: The Knowledge Dimension 


In the online classroom, the knowledge dimension relates directly to learning, which is about bridging the 
learning gap through well-designed instructional materials. According to Gagne (1965) learning is process that 
changes human disposition over a period of time and which cannot just be attributed to general human growth. 
According to Knowles (1984) learning is the process of gaining knowledge and expertise. Learning is therefore part 
of the human experience that involves intentionality to bring temporary or permanent change in the human behavior 


and cognition. 


As part of their college entrance requirements, students from diverse linguistic background have 
demonstrated English language proficiency and have taken one or more standardized tests in English. For 
undergraduate students typically in addition to the usual Scholastic Assessment Test (SAT) or American College 
Testing (ACT), they have had to provide proof of English language proficiency as measured by scores on the Test of 
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English as a Foreign Language (TOEFL). Similarly, graduate students are expected to pass the Graduate Record 
Examinations (GRE), Graduate Management Admission Test (GMAT) or other professional tests and the English 
language proficiency test TOEFL. Additionally, these students possess a level of conversational skill that allows 
them to communicate with peers and instructors, but it is academic language and the rhetorical conventions of their 
discipline that provide unique challenges because of content-specific vocabulary, complex sentence and syntactical 
structures used to express complex thoughts and concepts. For example, the language of literacy in science, which 
includes analysis of research is different from that of history, where there is discursive practice with discussion and 
exposition of historical events. 

Two factors that pose difficulties for these English language learners are related to content rigor and 
cognitive load of the learning tasks. Diagram 1, adapted from the work of Cummins (1984) lays out the relationship 
between the cognitive load of tasks and language support. Task that are cognitively easy and require less language 
support are easiest for the linguistically and culturally diverse learner, while tasks that require cognitive effort and 
much language support are difficult as both dimensions present learning challenges. 


Low cognitive load tasks 


Tasks that are easy, but require 
much language support 


Tasks that is both easy and require 
less language support 


e Course communications that might 
contain idiomatic expression, 
figurative language or complex 
sentences. 

e reading text not supported with 
images, illustration 


¢ Content readings that matches the 
current level of learner’s 
proficiency 

e Simple written tasks, such diagrams 
& charts, simple written responses 
in course discussions 


Lot of language 


Less language 
support needed 


support needed 


Tasks that are hard, but require Tasks that are both hard and 
much less language support require much language support 
e Science experiments e lectures without visuals 
e Math computations ¢ explanations of abstract concepts 
¢ Complex texts, but supported © writing academic papers 
with illustrations, images ¢ conducting and writing research 


High cognitive Load tasks 


Diagram 1. Learning Task and Learning Support Relationship (adapted from Cummins (1984) 


Becoming aware of the hurdles that learners face can help when creating learning materials and providing 
help with the learning process. Content rigor relates to the intrinsic rigor associated with learning the content of a 
particular discipline. In order to provide guidelines for understanding text complexity, the Common Core State 
Standards (2009) provided three specific areas to consider when determining text complexity: quantitative features, 
such readability measures; qualitative features, such as the language used, the levels of meaning, and knowledge 
demand, and reader/text relationship factors, such as the reader’s motivation, background knowledge, and task 


422 


variables that include the complexity associated with the task. While the Common Core measures of text 
complexity are applied most frequently to K-12 texts, and while the measures that are in place for gauging text 
complexity in college level textbooks are typically readability formulas, such as Flesch scores, it can be argued that 
the three dimensions: quantitative, qualitative and reader/task could be applied to textbooks at the college level to 
provide a more complete understanding of text complexity. 
In order to deal with complex text and demanding tasks, the workshop broke down the two areas 
suggesting interventions that the subject matter experts/course faculty could do and those that the instructional 
designers designing online classes could do. 


Interventions for addressing knowledge dimension 


Content related issues for learners 
who are linguistically and culturally 
diverse 


What can subject matter 
experts/course faculty do when 
organizing content for the 
instructional designer? 


What can the instructional designer 
do during the design and 
development phase of the course 
with the content? 


Content Rigor 


Provide support with reading and 
writing tasks 

> Address discipline specific 
academic vocabulary, 

> Use chapter summaries as advanced 
organizers 

> Provide chapter outlines 

Writing support 

> provide samples of academic 
writing that provide good models of 
language use, structural aspects, 
and discipline-specific rhetorical 
patterns 

» provide writing support through 
applications such as Grammarly 
which addresses grammar rules, 
contextual spell checking, 
vocabulary enhancement, 
plagiarism checker, and provides 
suggestions as well as corrects 
writing mistakes 

provide writing style support for 
APA, MLA, Chicago or other 
stylistics with information from 
robust internet sites such as OWL 
Purdue. 


Design digital learning objects in 
online courses that address 
Vocabulary through interactive 
vocabulary banks 

Provide visual support for complex 
information from the text 

Use Infographics and other learning 
aids to provide advanced 
organizers, vocabulary support, 
chapter summaries, microlearning 
videos, digital story telling related 
to the course content using 
narrative design frameworks. 


Cognitive load of tasks 


Reduce intrinsic cognitive load 
Activate prior learning 

During learning plan to use 
intermittent learning checks, reduce 
academic rigor of content-specific 
vocabulary and complex sentence 
structure. 

Implement activities that encourage 
self- check, use application and 
extension of learning through 
problem-based learning or real- 
world experiences 


Design digital learning objects in 
online courses that 

follow a learning path derived from 
course objectives, assessments, 
instructional strategies that breaks 
up the content but is in sync with 
other course materials, such as the 
text or other learning aids. 

enrich course materials with visual 
design that promote learning 
through proper layout design. 


Pedagogical orientation 


Be more teacher directed even 
when the orientation is more 
constructivist. 


Design with culture in mind 
Essential reflective questions: 
Will the students get this? 
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Make place for direct instruction, How can we design this based on 


not necessarily lecture. what I know and understand about 
Build cooperative learning the culture? 
activities 


Be open to intercultural learning 
and to other cultures: 

Views about time and space 

Views and expectations from 
teachers towards teacher 

Attitude toward different kinds of 
learning tasks 

Attitude toward the learning context 


In order to be able to design effective and engaging instruction both subject matter experts/course faculty 
and instructional designers have a key role to play in creating courses where learning is made accessible through 
consideration of cultural factors that impact learning. International students face unique academic challenges while 
at American universities despite having been academically successful in their own countries. The challenge of 
academic language and acculturation create stressors that can be mitigated by culturally sensitive online pedagogy 
and course design, where a conscious link is established between culture and learning. 
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Abstract: 


The Design4Online faculty training program is a successful training framework for online teaching faculty. 
The participants are instructors from different departments and colleges who teach online courses in ICON-a 
learning management system at the University of lowa. Design4Online includes three main components that 
encompass instructor learning, course development, and community building. The eight-week training program 
utilizes the ADDIE model, Community of Inquiry (Col) framework, Quality Matters (QM) rubrics, and data-driven 
research to ensure an effective development process, meaningful learning experiences, and training quality in the 
online educational environment. The Kirkpatrick Evaluation model was used to evaluate the training sessions and 
the overall training program. Both quantitative and qualitative methods were utilized to collect data from the 
participants throughout the training process to improve the quality of the training program throughout its initial 
implementation as well as future iterations. 
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Introduction to the Training Program 


Design4Online (D40) is a faculty training program created by the instructional design (ID) team at 
Distance and Online Education at the University of Iowa. This comprehensive program assists instructors at all 
experience levels in designing or redesigning online courses through the implementation of best practices and 
leading online pedagogy. The overall goals of the program are to empower instructors to use data-driven and 
student-centered online teaching strategies, to design motivating course materials and media, to create engaging and 
accessible student learning experiences, and to align their courses with national quality assurance standards for 
online learning. 

The D40 training program of 2018 consisted of four, face-to-face (F2F) training sessions and four online 
modules that enhanced and expanded the F2F meeting content. The participants had opportunities to complete skill 
development and course development exercises and to join one-to-one consultation meetings with the instructional 
designer for their online course development. Following the F2F and Online training components, the participants 
were encouraged to join and engage a community of practice titled “Community of Learners’. This community of 
learners serves multiple goals, but the predominant ones included creating space for continued growth and support 
of online learning, to share and learn data-driven research and best practices for online course design, development, 
and delivery. 


Training Program Background 


Distance and Online Education (DOE) at the University of Iowa offers students from across Iowa and 
around the globe the opportunity to attend distance and online programs. DOE’s primary task is to collaborate with 
faculty to create effective online courses implementing best practices and online pedagogy that encourage transfer 
and retention of knowledge. Over the past two academic years, instructional designers at DOE have led the design of 
more than 239 new semester-based courses and course revisions. Enrollment during those academic sessions has 
grown consistently. During the Spring of 2016, a total of 12,909 students enrolled in online courses at the University 
of Iowa. This was up 26% from 10,249 in Spring 2014 (DOE Internal Enrollment Source, 2016). 

As the enrollment numbers for online courses increase, it became imperative that DOE provide instructors 
knowledge, practice, and support in the development and implementation of quality online courses. To address this 
need, the instructional design team created the Design4Online program (D40) to provide online instructors with 
data-driven research and best practices for their online course design, development, and delivery. With the support 
of Distance and Online Education and the University College at the University of Iowa, the first Design4Online 
program was created in early 2017. 


Methodologies 


The D40 Training Program provides participants with opportunities to explore and engage with leading 
methodologies, frameworks, research, and best practices regarding online pedagogy. With D40O being the first 
training program at DOE, attention and dedication to finding seminal, current, and impactful data as well as 
effective practice/strategies was key. The ADDIE model, as cited in Kurt, 2017, was selected to guide the program 
development process as it is a widely used and respected framework for designing and developing educational and 
training programs. ADDIE can be broken into the following areas: Analyze, Design, Develop, Implement, and 
Evaluate. A strict linear process is not required by this model as it recognizes the need for the program development 
to be iterative. This approach is necessary as it clearly defines key stages which enables the ID to implement 
effective training tools to meet the goals of the training program (Kurt, 2017). 

In addition, the IDs worked with the participants in using the Quality Matters (QM) Rubric, which is the 
leader in providing quality standards for online and blended courses. The QM Rubric helps to ensure online course 
quality and participants’ success. Moreover, Adult Learning Theories, Active Learning Methods, and the Community 
of Inquiry Framework were applied to the training sessions with the expectation that the participants would be able 
to adapt and apply what they had achieved from the training program to their online teaching. Through the hands-on 
training sessions, F2F group activities, and online activities and discussions, the participants were able to simulate 
their online teaching activities. 
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Training Program Format and Process 
The program format consists of three phases as listed below: 


Phase One: Training Effective Online Course Design 

The first phase provided training regarding effective online course design. Participants applied concepts 
learned throughout an eight-week training program to either the design and development of a new online course, or 
to the revision of an existing Distance and Online Education offering. The program included both face-to-face and 
online learning training. Trainings provided “How To” instructions, demonstrations, and modeling regarding best 
practices for online learning. It reflected the “I Do, We Do, You Do” gradual release model (Wheldall, Stephenson, 
and Carter, 2014) that employed practical application of the best-practice strategies. It also included data-driven 
research from adult-learning theory, multimedia learning theory, and the cognitive theory of multimedia learning. In 
addition, it implemented the backwards design process, peer-conversation, and reflection, as well as networking 
platforms to establish a community of practice around effective online learning. 


Phase Two: Design and Development Support 

The second phase continued to support instructors through the completion of implementing their online 
courses that were begun in phase one. It was comprised of individualized meetings with instructional designers as 
well as support from other team members such as media and exams. 


Phase Three: Building a Community of Practice 

The third phase was directed to building a community of practice. It included the instructor participating in 
three of the five community of practice engagement activities led by the instructional design team. The activities 
included the option to engage in face-to-face coffee shop discussions or online zoom meetings. 

The program format was established through implementing the ADDIE Model to guide our process. The 
ADDIE Model is an iterative model that has five stages: Analysis, Design, Development, Implementation, and 
Evaluation. 


1. Analysis Stage: To begin the development of the training program, a needs analysis survey was created and 
sent out to all online instructors at the University of Iowa. The purpose of the analysis stage was to get 
initial information from the instructors to create the training topics, timelines, and address facility needs for 
the training program. The ID team designed the survey and sent it out to the instructors in the Spring 2018. 
171 responses were collected. The results from the initial survey could be summarized as the followings: 

The number of faculty: 147 (Faculty: 82% and Adjunct: 16%) 
- Others: 24 
-  Departments/Colleges: Wide spectrum of departments/colleges and programs at the University of Iowa 
- Online teaching experience: 25% 
- No experience as an online student: 69% 
- ‘Traning Topic Interests included: 
Alignment 
- Assessments 
- Instructional Strategies 
- Course Management 
- Evaluation 
Engagement 
- Pitferente Training Time: 1:00-4:00 on Thursdays 


The training topics of interest were then broken into four specific training sessions that included two-week 
supplemental, online modules. 


2. Design Stage: Since this was a new program and required a lot of preparation, the team then divided up into 
the following sub-program areas: Program Branding and Marketing, F-2-F Training Framework, Online 
Session Framework, Logistics, Evaluation, Certificate, and training content. Each ID member was in lead 
of working on one or two of the areas for the program. The ID team also chose an overall program lead that 
would oversee the entire program and sub areas. The ID team worked collaboratively and effectively to 
make the program work effectively. Each of the sub-program areas was detailed as follows: 
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Program Branding and Marketing 

Worked with the Marketing Section at DOE and UI to advocate the D4O Training Program. 
Created a cohesive visual brand for D4O, created a D4O web presence, and created several print and 
email marketing materials. Although this was the first time to market the program, it was successful in 
creating brand awareness. 


F2F Training Framework 

Designed the F2F framework (e.g., lesson plans, agendas, teaching and learning activity 
supports, and methodologies). All the designed templates for the F2F session were presented to the ID 
team for feedback. The final template was consistently used for all sections throughout the D4O program. 


Online Session Framework 

Designed an online course for the D4O Training Program in ICON. The course structure, all the 
online module pages, and assignment page templates were presented to the ID team for feedback. The 
final page templates were consistently used throughout the D4O training program. The F2F framework 
and the Online Framework lead instructional designers collaborated together to make sure participants had 
a smooth transition from the F2F training session to the Online course site, as well as an overall cohesive 
experience that included a branding feel, topical connections and enhancements. 


Logistics 

Worked with sections in the DOE and other related departments at UI to accommodate budgets, 
application process, registration, facilities for the training program, recognition items, and other details for 
the D40 Program. 


Evaluation 

Established an evaluation process to ensure program training quality. The Kirkpatrick Evaluation 
Model was chosen to guide this process. It began with a pre-program survey that was sent out to the 
program participants to measure their general prior knowledge about each of the upcoming training topics 
as well as their general knowledge and experience with online pedagogy. 

At the F2F training sessions, a Choose A Spot Activity was implemented at the beginning of the 
training session to measure the participants’ knowledge to the specific training topic. At the end of the 
F2F training session, a post-survey was distributed to the participants to measure their gained knowledge 
on the training topic. 

After the F2F training session, expanded and enhanced learning took place in the two-week 
online course. After each participant completed the online module, they were asked to complete the End- 
of-Module Survey. The survey helped the participants to assess their ability to apply what they had learned 
in each specific topic area. To exit the D4O training program, the participants were asked to take part in a 
Post-Program Survey to measure their achievement. This opportunity encouraged the participants to self- 
evaluate their individual progress and assess if they met the training goals. The survey results helped the 
IDs to evaluate the strengths and weaknesses of the D4O training program. It allowed for effective 
training adjustments as the program unfolded, as well as provided the IDs with feedback regarding 
necessary changes for future iterations of the training program. This has allowed the IDs the ability to 
maintain and improve the overall training quality of the D4O Training program. 


Certificate 

Designed and reviewed by the ID team as a way to congratulate and honor participants for the 
dedication to improving online courses. It was approved by the Associate Dean of Distance and Online 
Education. The certificate was signed by the Associate Dean of Distance Education and Outreach and the 
Associate Provost for Undergraduate Education and the Dean of the University College. The certificate 
was presented to the successfully completed program participants at the banquet. 


Training Content 

The D40 program provided the four following training modules: Alignment, Assessments and 
Instructional Strategies, Engagement, and Course Management and Evaluation. Each of the training 
topics were designed, developed, and delivered by two IDs. Each of the training topics were reviewed by 
the ID team and revised upon the ID team feedback. The IDs for each of the training topics rehearsed the 
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presentation with the ID team and other staff at DOE for feedback and made any necessary revisions prior 
to the actual presentation. 


Development Stage: At this phase, the IDs used the data gathered from the earlier phases to create the 
training program. Each sub-program area lead took on the tasks of drafting, reviewing, producing and 
evaluating their specific areas content for the F2F training meetings, online modules, and the ID consult. 


Implementation Stage: Design4Online training program has three main components that encompass 
instructor learning, course development, and community building. an 8-week hybrid course. The course 
included five bi-weekly face-to-face meetings where instructors and the DOE instructional designers 
addressed the six training topics. These three-hour meetings were complemented by required two-week 
online modules that enhanced and broadened the face-to-face content and provided activities for instructors 
to implement effective online pedagogy. The training content included research and design reflecting the 
Community of Inquiry Framework (Garrison, Anderson, & Archer, 2000) that addressed the critical 
learning occurring as experiences intersect between social, cognitive, and teaching presences. 


Each face-to face training began by assessing the participating instructors’ reaction to the training 
topics of that day via an engaging “Choose a Spot” activity. The activity provided peer support and 
encourages awareness, discussion and collaboration regarding the day’s topic/s. The training session 
implemented the explicit instruction, gradual release model “I Do, We Do, You Do” (Wheldall, 
Stephenson & Carter, 2014) to successfully engage and support the participants in the online training 
content. 


The online environment had specific modules for each face-to-face training session that enhanced and 
built upon the face-to-face training and collaboration with peers. It also provided each participating 
instructor deeper and more meaningful application of the knowledge. 


1-1 ID Consultations involved the participating instructors and ID who provided individual course 
consultation. The ID and the instructor met on a weekly basis to support and transfer the learning of 
the face-to-face content and online modules to an instructors’ upcoming online course. 


Learning Community was developed to support the participating instructors. After the training was 
completed, the participating instructors were invited to join a collaborative community of practice. 
This community included the instructors and the instructional designers who continued to explore and 
create leading online pedagogy, practices, and research. The activities will include the option to 
engage in face-to-face coffee shop discussions or online ZOOM meetings. 


Evaluation Stage: Evaluation was an iterative process. As mentioned above in the Evaluation area in the 
Design Phase, the D4O training program provided the participants many survey opportunities to evaluate 
their prior knowledge to the training program, during the training sessions, and after the program 
completion. The survey results helped the D4O Training Program evaluate strengths and weaknesses of the 
training program in order to maintain the training quality and to improve the training sessions and the 
whole training program for the next training cohorts. 

The Kirkpatrick Evaluation model was utilized to design the surveys for the participants to measure 
their Reaction, Learning, Behavior, and Results regarding the D4O Training Program. The surveys were 
used to assess the participants levels of confidence regarding each of the training topics. The surveys used 
the following terms for measurement: Very Confident, Confident, Undecided, Not Very Confident, and Not 
at All Confident. The table below is a summary of those results. The first column indicates which of the 
four levels of criteria within the Kirkpatrick Evaluation Model is being measured. The second column 
indicates the percentage of participants who were either “Very Confident” or “Confident” prior to the 
training. The third column indicates the percentage of participants who were either “Very Confident’ or 
“Confident” following the training program. Table 1 shows the confidence levels about the training topics. 
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Levels Pre-Survey Results Post Survey Results 


Reaction 17% 94% 

Learning 41% 92% 
Behavior 13% 93% 
Results 20% 54% 


Table 1. Confidence Levels 


The following information will use the use the term “confidence level” to reflect the percentage of 
participants who indicated in the surveys that they were either “Very Confident” or “Confident” about each level of 
the Kirkpatrick Evaluation Model criteria. As an overview, the survey results from the pre/post program surveys 
showed that the confidence levels of the participants about the training topics increased sharply in each level of the 
Kirkpatrick Evaluation Model criteria. The Reaction criteria began at a 17% confidence level. After the training 
program, the confidence level grew to 94%, an increase of over 70%. The participants indicated prior to the training 
a Learning confidence level of 41%. The post-training survey showed the participants Learning confidence growing 
by over 50%, with a 92% confidence level score. The Behavior confidence level prior to training was 13%. The 
post-training survey Behavior confidence level was 93%, an 80% increase in confidence. The final criteria, Results 
was a bit more difficult to measure. Results criteria measures what participants accomplish post-training. The 
Design4Online had as part of its process, the development of a future online course by each of the participants. At 
the time of this training program report, not all future courses had been implemented, and many of the courses had 
not yet begun. Therefore, only partial data has been gathered regarding the Results level criteria. However, the 
current measurement for Results includes the pre- training level of 20% and a post-training level of 54%. This is a 
growth of 34%. This increase is a positive indicator of the final Results growth percentage. The final Results level 
will be measured and evaluated as the courses continue to be implemented and the data gathered. 

In order to support the survey data, qualitative data from the participants through interviews, takeaways 
from the showcase, notes from the training program, and open-ended survey items were also collected and analyzed. 
The data were triangulated, categorized, and reflected the four levels: Reaction, Learning, Behavior, and Results. 
Below are some quotes from the participants: 


Reaction: Nine (out of eleven) participants stated that the training program and content were “extremely 
useful, engaging and intensive, and incredible.” One of the participants indicated, “the instructional 
designers at DOE have really high standards; they will not allow you to end up with a bad product. They set 
the bar extremely high and inspire you to do your best.” 


Learning: Nine (out of eleven) participants mentioned that they learned a lot from the D40 training 
program. One participant, for example, expressed, “I learned so much that I didn’t know before about the 
importance and value of alignment, varying assessment and instructional strategies...” 


Behavior: All of the participants indicated to apply what they had learned from the D40 program to their 
online courses. For example, one participant said, “Participation in D4O changed my perspective of 
distance learning...” Another participant also valued the training program and expressed, “Design4Online 
was an engaging and intensive program that has allowed me to greatly improve the quality of my online 
class.” 
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Results: All participants completed the eight-week training program. Some of the early comments from 
instructors included their desire to continue to worth the instructional designers to build their course fully 
implementing what they have learned in the Design4Online training program. One participant expressed, 
“Thank you all so much for this wonderful course! It has completely broadened my horizons not only about 
online teaching but teaching more generally.” 


Conclusion 


Over the past two academic years, the enrollment for online courses at the University of Iowa has grown 
consistently. The instructional designers at Distance and Online Education (DOE) recognized this growth and 
created a training program, Designe4Online, to assist instructors by providing knowledge, practice, and support in 
the development and implementation of quality online courses. The Design4Online Faculty Training Program is the 
first of its kind at the University of Iowa. It is based on the ADDIE Model and includes data-driven research and 
best practices for online course design, development, and delivery. It is also implemented Adult Learning Theories, 
Active Learning Methods, and the Community of Inquiry Framework. The Kirkpatrick Evaluation Model provided 
an evaluation method for each of the training modules, the training process, and the overall training program. 
Quantitative and qualitative data were collected from the participants, analyzed, and reported in order to make any 
necessary changes to the content, the course, and the program for next training programs. The participants of the 
training program have shown great success in learning, applying, and providing results that indicate their ability to 
confidently and positively develop effective online courses has increased. Overall, the evaluation criteria of the 
Kirkpatrick Evaluation Model has grown at no less than 34%, with some areas growing over 50%. In addition, 
participants shared how much they enjoyed and appreciated the training program. One of our participants stated, “I 
loved it. I learned so much. I can’t wait to recommend it to all my faculty friends.” 
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Abstract 


Augmented reality (AR) as an emerging technology has gradually been introduced into educational 
contexts, and previous research presented promising results of AR involved curriculum and class activities. This 
study incorporated an AR book set to six pre-kindergarten classrooms. Students in the three experimental classes 
engaged with an AR picture book while the students from three control classes only accessed a 2D picture book of 
the same content. The results revealed that students in experimental classrooms had significant growth in letter 
recognition and motivation compared to their peers in controlled classes, which implicates the effects of AR in areas 
of letter recognition and motivation within the preschool literacy learning process. 


Introduction 


Reports have shown augmented reality (AR) has been described as a newly emerged technology that 
combines computer-generated virtual information into the real environment (Azuma, 1997; Di Serio, Ibafiez, & 
Kloos, 2013). However, most of the AR related research has been focused on grades beyond those in which children 
learn to read and STEM related fields (Yuliono, 2018; Sirakaya & Alsancak, 2018), AR’s effect on young children 
especially on their literacy learning is not known. To merge the gap, this study integrated an AR picture book into 
the centers/stations of kindergarten classroom and examined AR’s impact on pre-kindergartener’s literacy 
development as well as motivation in literacy learning. 


Literature Review 
In the field of learning technologies, augmented reality (AR) has begun to draw attention from many 


educators and researchers (Akcayir & Akcayir, 2017; Huang, Chen, & Chou, 2016); numbers of previous studies 
have indicated that AR can enhance students’ learning performance (Chang, Hou, Pan, Sung, & Chang, 2015; 
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Ferrer-Torregrosa, Torralba, Jimenez, Garcia, & Barcia, 2015). For instance, by incorporating AR into lessons, 
students have been known to exhibit a higher degree of curiosity and surprise (Bujak et al., 2013) a better 
understanding of the content they are learning (Yoon, Elinich, Wang, Steinmeier, & Tucker, 2012); and more 
interest in reading storybooks (Wang, Lee, & Ju, 2019) . Moreover, studies have also shown that AR can make 
learning more attractive and effective (Diinser and Hornecker, 2007) and can enhance students’ attention span as 
well as learning motivation (O'Brien & Toms, 2005; Sumadio & Rambli, 2010). 

Despite the development of a range of AR integrated learning materials (McKenzie and Darnell, 2004; 
Woods et al., 2004; Saso et al., 2003; Zou et al., 2004) and a growing number of related research initiatives that have 
been conducted (Brown & Harmon, 2013), studies that involve young children (3-6 years old) are scant (Akgayir & 
Akgayir, 2017; Sirakaya & Alsancak Sirakaya, 2018). Moreover, an interesting distinction in the body of literature 
on Augmented Reality is that most studies use marker-based as opposed to location-based AR meaning the 
interactivity is triggered by scanning an image rather than moving about the environment (Sirakaya & Alsancak 
Sirakaya, 2018). 

In this research, an AR integrated picture book was introduced to young children (3 to 6 years of age). By 
scanning each page of the picture book using an application on tablet screens, students engaged and interacted with a 
3D animation that delivered vivid multi-sensory content about letters and vocabulary (sound and visual stimuli). 
Based on previous literature, reading picture books has been proven to be a promising learning activity for language 
and literacy skill development (Bus, van IJzendoorn & Pellegrini, 1995; Lever & Sénéchal, 2011) as well as 
cognitive engagement and artistic thinking (Cheng & Tsai, 2014; Elia, van Den Heuvel-Panhuizen, & Georgiou, 
2010; Hsiao, 2010) in an early childhood educational context. Interestingly enough, a research initiative led by 
Masataka (2014) showed that given two sets of picture books with the same content, the e-picture book paired with 
touch screens lead to better learning performance than the print version. Furthermore, Diinser, Walker, Horner, and 
Bentall (2012) found that AR enhanced books are more effective than traditional books with text and illustrations in 
secondary physics learning. Building upon previous research findings and to expand the scope of prior 
investigations, this research will study the effects of an AR incorporated picture book on young children’s language 
art skills, and it will address two research questions: 

(1). Will access to augmented reality (AR) activities impact young children’s (3-6) letter recognition ability? 
(2). Will access to augmented reality (AR) activities impact young children’s (3-6) motivation to acquire literacy 
skills? 


Methods 


Participants and Context 

Three early childhood schools located in the southwestern region of the United States participated in this 
research. This study followed a cluster randomized research design that involved three experimental classes and 
three control classes from the three schools. In each school, two pre-kindergarten classes were chosen, one class 
served as an experimental group that used an AR-enhanced picture book, while the other class participated as a 
control group utilizing the very same electronic book minus the AR features. Daily schedules across all campuses 
included: self-directed activity time, whole group instruction, play time (mostly outdoor recess), and mealtimes. 
While all schools had the same types of activities each day, the order of the activities varied. This selection 
approach is meant to control the potential variance associated with different educational environments at each of the 
three sites. 

Teachers often had at least one assistant in the classroom at all times and there was a general emphasis on 
letting children learn through play. Many of the centers at all sites involved activities that allowed children to build, 
create, read, and engage in pretend games. The observer notes showed that these classrooms were student-centered 
and often activities were led by student initiative except for whole group reading time or mandatory scheduled meals 
and outdoor activities. 


Augmented Reality Book Sets 

This augmented reality set comes with a picture book and an application on the iPad (disruptED, 2019). 
The contents of the book were based on the 26 letters used in the English alphabet with 2 letters on each page. The 
letters were displayed in color and had accompanying with cartoon figures that became animated in the AR version. 
As students scanned each page, animations were triggered on the screen of the iPad with music and narration to 
provide students a multi-media learning experience. The animation could be activated from different angles and as 
many times as students desired. Students were able to go to or jump to different letters simply by flipping the pages 
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of the picture book. At the end of the book, there is a final page which demonstrated all of the 26 letters, when this 
page was triggered, the letter songs will be triggered as a final review of all letters. 

There are tracing, letter matching (card reveal), and spelling games found throughout both versions of the 
book sets, and navigation through the AR book sets depended upon the version students were using. For example, 
with the AR version, turning the page could be done by swiping or scanning a particular page whereas the 2D 
version didn’t allow children to use the technology to interact with the book in the same manner, but presented the 
same games and content. 


Research Design 

In each of the three experimental classes, an AR picture book station was setup for six weeks (see Figure 
1); these stations consisted of an iPad, a timer, a picture book, and a roster. Each station was placed in an area of the 
classroom that students were already used to completing centers/activities in (for example, a round table). 


Figure 1. AR Station in One of Classroom 


Researchers are fully aware of the concerns and potential disadvantages of incorporating AR related 
activities into classrooms, especially in pre-kindergarten settings, since AR needs to be manipulated via technology, 
particularly touchscreens. A major issue found within current research is the fear of subjecting students to excessive 
amounts of screen time (Yilmaz, Kucuk, & Goktas, 2017), which may correlate with poor academic performance or 
health-related risks due to reduced time sleeping and exercising (OECD, 2015; Park, Kang, & Kim, 2014). To 
mitigate the impact of these risks, this study intentionally designed our approach of integrating an AR picture book 
into early childhood classrooms through systemic time limitations. In each class, students were able to interact with 
an AR picture book station during self-directed learning blocks, but they were only be able to use the materials a 
maximum of 10 minutes each turn or session. A timer was provided to avoid excessive screen usage, and children 
were only allowed to visit the station once or twice per day. Under this strategy, this study attempted to protect child 
participants from potential harm brought by overusing the technology, and therefore create a safe space for students 
to explore and participate in AR integrated picture book activities. Therefore, students came to the station anytime 
when their teacher allowed self-directed activities and interacted with the materials for up to 10 minutes, up to three 
times per day. A timer was left on the table to limit the time spent to avoid the excessive electronical devices usage. 
Researchers assisted teachers keeping track of the time by allowing students to mark their names on a roster. 
Students then left the table when their time was up and moved to other stations. During the process when students 
were interacting with the AR picture book, researchers took field notes, which documented students’ interaction 
process and behaviors of students interacting with the technology. 
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Beyond the concern for screen time and student safety, attention must be paid to whether or not students 
(and teachers) are able to take advantage of the AR materials in their entirety. Uygur, Yelken, & Akay found that 
only half of the teachers participating in their study had ever heard of AR and “78.2% (172) of the teacher 
candidates stated that they never used the augmented reality applications before” (2018). Their recommendations for 
the future use of AR by teachers is they receive appropriate training and exposure to AR technologies in order to 
grant users more affordances via deeper knowledge. In order to ameliorate this issue, each teacher received training 
prior to their students’ use of the materials and was visited at least once during early implementation to ensure 
students were able to take turns using the device. 


Assessment 

All of the six participating classrooms participated in both pre- and post-assessments delivered by 
researchers before and after the intervention, the same assessment was used for both pre- and post-tests. Students 
were asked to identify as many letters as possible from a PowerPoint to determine if there were any gains in rapid 
letter naming throughout the study. On each slide of the PowerPoint, students were shown a letter for up to five 
seconds, and researchers then recorded correctness of letter recognition within the timeframe. In order to attain a 
higher level of alignment between the pre-test, post-test, and the book sets, the assessments were created in the same 
font as the materials students were using. 

Teachers from the participating classes were asked to fill out a student motivation survey that included a 
smaller set of questions from the larger CLI Circle Engage assessment (UTHealth, 2018) (see Table 1), and were 
asked to participate in an interview before and after the study that sought for their opinion on the AR picture book in 
terms of its affordance upon students’ literacy achievement and growth in motivation. Teachers both made 
predictions and reflections about the use of AR in early childhood literacy during these interviews indicating that 
their views on AR might have evolved or changed. 


Table 1. Assessment of Motivation to Read 


Motivation to Read 


Please rate the following statements 1 - Rarely 3 - Consistently 


Child shows interest in reading by self-selecting books during 
centers or free choice periods. 


Child shows enthusiasm and engagement during shared or interactive 
reading activities. 


Child asks to be read to (including books and/or print in the 
environment). 


Child asks the meaning of text (including books and/or print in the 
environment). 


The data source was first organized based on the scores of pretests and posttests by the control groups and 
experimental groups. Independent t-tests were conducted to compare any differences between the two groups. 
Teachers’ interviews were transcribed and coded by the researchers along with field notes following a bottom-up 
scheme (Miles & Huberman, 1994). 


Results 


Letter Recognition 

For experimental classes that involved AR picture books, the number of letters students in these classes 
could recognize in the posttest was not significantly different from their pretest (¢ = 1.02, p > 0.05). There were no 
significant differences in students’ posttest between experimental classes and control classes (t = 0.9, p > 0.05). A 
possible reason could be attributed to students’ high prior knowledge about letters, because teachers mentioned in 
interviews that based on their teaching experience, the average number of letters students in this age group can 
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recognize should be around 13, while the pretests showed that students in these three schools were able to recognize 
are 20 out of 26 letters. However, the classroom observations showed that even though students from both 
experimental and control classes seemed to have learning growth, students in experimental classes appeared to have 
a larger jump in number of letters recognized. Thus, an independent t-test was used to test the differences of growth 
between these two groups’ pretest and posttest. The results showed that students from experimental classes had 
significantly more growth than the students from control classes (¢ = 2.36, p < 0.05). The results indicated that an 
AR picture book helped students in experimental group to recognize more letters. 


Motivation 

Data related to motivation was obtained via a Student Motivation instrument filled out by teachers. 
Students from all classes appeared to have high prior motivation in literacy learning. On average, their literacy 
motivation score was 2.39 out of 3 in the pre-test. The results from the t-test indicated that students’ motivation in 
experimental classes increased significantly (¢ = 2.32, p < 0.05), while the score of control group was not. However, 
there was no significant difference between the two groups’ posttest scores. These results suggest that AR can 
increase students’ motivation during literacy learning, as the score in the experimental group increased significantly. 
It is possible that the 2D video can also improve students’ motivation, which may explain why the posttest scores of 
the two groups were not significant. 


Discussion 


The goal of this study was to examine the effects of augmented reality on young children’s letter 
recognition ability and motivation to practice and learn literacy. Students in the control group and experimental 
group used the same application with the same content and activities embedded except the control group used a 2D 
format while the experimental group used AR. A main feature brought by AR is its movability; by scanning each 
page of the picture book, a 3D animation corresponding to a specific English letter would pop up. Students were 
then able to interact with it by touching the screen, and/or by involving kinetic movements such as to look at the 
letters from different angles and distances. This spatial interaction could not be obtained via 2D animations in which 
students can only manipulate the learning contents via the screen itself completely disconnected from the physical 
book. As a teacher stated in the post-interview, this application provided kids: 

“with an opportunity to interact with letters in a different way than they have right now. So, I think it will 
definitely add a layer and add a different dimension into classroom.” 

According to Sawyer’s (2005) theory about learning science, an important approach to enhance the learning 
process is to implement strategically regulated repetition, which means contents need to be exposed to learners 
repeated but via diverse vehicles and formats. In the 2D animation, when students flip to a new letter, they first saw 
the image of the letter, then they heard the pronunciation of the letter in a word, which created a multisensory 
experience. However, AR features added a more dimensions and layers of repetition relying on children’s spatial 
perception in virtual environments; students were now able to view the picture book but also the virtual animated 
letters from multiple angles and distances (see Figure 2). This feature helped students in the experimental group 
learn more letters and be more highly motivated, since according to the teachers, this layer of repetition is “both fun 
and education, which was a treat for us.” 
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Figure 2. Students interact with the AR book sets 


Different from other AR research for older age groups and different subject areas (Cheng & Tsai, 2014; 
Lever & Sénéchal, 2011). the learning outcomes about the effects of AR in this study were not as salient. Teachers 
from both experimental and control groups in their interviews have mentioned that 2D animations and AR features 
all provided intriguing and attractive multi-media environment to students that can foster their learning 
performances and motivation. Both versions were presented with vivid animations and narration in a self-paced 
manner, which is beneficial in cultivating meaningful multimedia learning (Mayer, 2002). This explains why the 
post motivation scores of experimental groups were significantly different from pretests while not significantly 
different from control groups posttests. These findings provided implications for promoting the designing AR 
incorporated educational applications. Although AR features grant students an innovative way to revisit the learning 
content, the effectiveness of AR might vary based on the nature of the content. For learning more complex contents, 
AR’s potential in enhancing students’ learning would be better manifested. Specifically, the authors of this paper see 
a potential for future research to investigate more geometry and science-related content that can take more 
advantage of the difference between the 2D and 3D capabilities of AR picture books. 


Limitation 

One of the limitations in this study is student sample; this group of students, based on pre-tests, had shown 
a high prior knowledge on their letter recognition preassessments, which was also confirmed by their teachers via 
interviews and surveys. On average, during interviews, teachers estimated that children in PK were able recognize 
13 letters at the time of the study in spring 2018. However, the students in this study were found to recognize 20 
letters. The small range of measurable growth might mask a potentially significant research outcome. In the next 
stage of research, we are looking for enrolling students with more diverse levels of prior knowledge in letter 
recognition abilities or investigating other content altogether. Another potential limitation is teacher bias during 
interviews or surveys due to the infeasible direct data collection from young children, thus the motivation data was 
collected through teachers’ perspective via the form of Likert-scale questionnaires. Furthermore, when conducting 
the study, we noticed that some data may not represent special needs students in a fair manner; particularly students 
who may have speech production or visual impairments. 
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Abstract 


Amid the continued growth of online learning—and concerns about its quality—a number of different 
groups have moved to establish tools, such as rubrics and standards for online course quality. This paper highlights 
the development of the Instructional Design Standards for Distance Learning by the Association for Educational 
Communications and Technology. AECT is the leading international professional association for the scholarly study 
and practice of instructional design. Also featured is a comparison with other popular tools and suggestions for use 
of the AECT Standards 


Introduction 


With approximately 30% of college and university students nationwide enrolled in one or more online 
courses, it is safe to say that distance learning has become fully institutionalized into the fabric of U.S. higher 
education (Pifia, 2008; Seaman, Allen & Seaman, 2018). In spite of the national trend of decreased enrollments at 
colleges and universities during the past few years, enrollments in online courses have continued to increase 
(National Center for Education Statistics, 2019). 

Notwithstanding the growing ubiquity of distance learning, and an ever-increasing body of research 
indicating that students can learn well online (e.g. Means, Bakia & Murphy, 2014; Means, Toyama, Murphy, Bakia 
& Jones, 2009), skepticism regarding its quality persists. Opinion polls continue to report that many postsecondary 
faculty members feel that learning online is inferior to learning in a traditional classroom (Jaschik & Lederman, 
2018). 

Advances in distance learning research and practice, and efforts to address the elusive concept of “quality,” 
have inspired professional communities, organizations and vendors to establish quality standards and rubrics for 
online courses. Standards “provide people and organizations with a basis for mutual understanding, and are used as 
tools to facilitate communication, measurement, commerce and manufacturing” (CEN-CENELEC, 2018, p.1). 
Popular distance learning standards and rubrics include: Quality Matters (Maryland Online, 2017), the Open SUNY 
Course Quality Review (OSCQR) rubric (Online Learning Consortium, 2018; Open SUNY, 2018); the OLC Quality 
Scorecard for Online Programs (Shelton, 2010), the INACOL Standards for Quality Online Courses (iINACOL, 
2011) and Blackboard’s Exemplary Course Program (Blackboard, 2017). 


AECT Instructional Design Standards for Distance Learning 


The Association for Educational Communications and Technology (AECT) is the most established 
international professional association for instructional design and technology (www.aect.org). It was established in 
1923 as the Department of Visual Instruction of the National Education Association and later rebranded as the 
Department of Audiovisual Instruction (DAVJ), with an initial focus upon the use of audio-visual technologies in 
classroom instruction (Reiser & Dempsey, 2017). As a result of evolution and progress in the areas of learning 
theory, communication technologies and an emerging field of instructional design, the Association for Educational 
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Communications and Technology emerged in 1970 as an independent professional association (AECT, 2001; 
Molenda, 2008). 

AECT launched the Journal of Instructional Development, the first scholarly journal dedicated to 
instructional design in 1977 (AECT, 2019) and the association maintains a position of leadership in research and 
practice in the field with its five journals: Educational Technology Research and Development, Tech Trends, the 
Journal of Applied Instructional Design, the Journal of Formative Design in Learning, and the International Journal 
of Designs for Learning. 


How the AECT Standards Came to Be 


During a series of discussions between AECT members and association staff, a recurring topic was that 
online courses at colleges and universities were being developed without the benefit of research-based instructional 
design guidelines. A small task force of AECT members researched and produced a first draft of ten instructional 
design standards for distance learning. An edited version of the standards was approved by AECT’s Executive 
Committee, by the board of its Division of Distance Learning, and by the association’s full Board of Directors 
(Harris, 201 7a). 

Once approval has been obtained by the association’s Directors, several leading AECT member scholars 
and practitioners were invited to write chapters providing evidence and justification for each of the ten standards. 
Those who contributed chapters were Saul Carliner, Yuan Chen and David Price (2017), Peggy Ertmer, Judith 
Lewandowski and Jennifer Richardson (2017), Phillip Harris (2017a; 2017b); Michael Molenda (2017), Gary 
Morrison (2017), Jennifer Morrison and Steven Ross (2017), Anthony Pifia (2017b; 2017c), Wilhelmina Savenye 
and Yi-Chun Hong (2017), and Michael Simonson (2017). 

Each chapter underwent a double-blind peer review process by a combination of faculty members from 
graduate programs in instructional design and technology and practicing professional instructional designers. Final 
corrected drafts of the standards chapters and a set of rubrics for their application (Harris, 2017b) were compiled 
into an edited book, /nstructional Design Standards for Distance Learning published by AECT (Pifia, 2017a). 
Members of AECT have access to a free e-book version of the book through the publications area of the AECT 
website (www.aect.org). 


What are the Standards? 


Standard 1: Purpose. Effective course design begins with a clearly articulated purpose. This is the 
standard to which all other standards must align. Purpose may be thought of as two-dimensional: institution or 
instructor and student. The design should include both the purpose of the course as envisioned by the institution or 
instructor and the purpose as viewed by the student. As the purpose is articulated through goals and objectives, 
collaboration between instructor and student will set a firmer foundation than can be achieved through a one- 
dimensional purpose statement. 

Standard 2: Assumptions. Course design must take into account assumptions that shape the purpose and 
subsequent course development. Most assumptions are based on students’ prior knowledge and established 
understandings and skills. Articulating these content assumptions provides a starting point for new learning. 
Assumptions in the case of online learning also encompass students’ ability to use delivery technology. 

Standard 3: Sequence. Learning opportunities must be sequenced in a manner that promotes efficient 
knowledge acquisition consistent with the prior-knowledge assumptions. Various models of sequencing—linear, 
spiral, scaffold, etc —should be considered, and the course design should incorporate those strategies best suited to 
the content within the constraints of online delivery. 

Standard 4: Activities. Learning is achieved through activities both passive (reading, listening, viewing) 
and active (experimenting, rehearsing, trying). Activities should be chosen that best suit the content, students’ levels 
of knowledge, experience, and ability, and online delivery constraints, particularly accommodating synchronous, 
asynchronous, and mixed course participation. Student self-selected or self-developed learning activities should be 
incorporated along with instructor-selected and instructor-developed activities, consistent with a two-dimensional 
purpose. 

Standard 5: Resources. A range of resources should be articulated to foster deep learning and extend 
course-centered experiences and activities. Resources should be multimodal to accommodate students’ interests, 
understandings, and capacities, consistent with course content and technological accessibility. Resources should 
allow students to go beyond the constraints of the formal course structure to engage in self-directed, extended 
learning. 
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Standard 6: Application. Consistent with providing for active learning, students should have integral 
opportunities within the course design to apply new learning. Effective course design incorporates opportunities to 
practice newly acquired understandings and skills, both independently and collaboratively. Online collaborative 
application opportunities should be developed using social media, and offline collegial groups also should be 
structured whenever physical proximity of students affords this opportunity. 

Standard 7: Assessment. Regardless of the model of sequencing learning opportunities, the sequence 
should include points of assessment for purposes of feedback and review, with instances of re-teaching as necessary 
for students to acquire full understanding. Formative assessment, whether formal, informal, or incidental, allows 
teachers and students to give feedback to one another and to review the operationalized design in order to revise the 
course design based on students’ input with regard to knowledge acquisition and effective use of new 
understandings and skills. 

Standard 8: Reflection. Effective course design must include opportunities for reflection as an extension 
of the Feedback/Review/Reteach standard. Reflection involves both instructor self-reflection and student self- 
reflection related to achievement of the purposes that have been articulated as the basis for the course. Such 
reflection is intended to deepen the learning experience and may serve as reiteration of purpose at key points during 
the course. 

Standard 9: Independent Learning. Effective course design incorporates opportunities for independent 
learning, both instructor- and self-directed. Online course development, particularly in the asynchronous mode, 
should epitomize independent learning, which should include opportunities for feedback, review, and reflection—all 
of which should resonate with the purpose. 

Standard 10: Evaluation. Course evaluation must be purpose-driven. Alignment with the purpose should 
be threefold: a) based on acquisition of new knowledge, understandings, and skills; b) based on instructor self- 
evaluation; and c) based on student self-evaluation. Multidimensional evaluation offers a fully articulated basis for 
judging the success of the course and the students as well as providing information that can help shape future 
iterations of the course. 


How the AECT Standards Compare to Other Tools 


The International Board of Standards for Training, Performance and Instruction (ibstpi), Maryland 
Online/Quality Matters, The Open SUNY Center for Teaching Excellence, the International Association for K-12 
Online Learning (INACOL) and Blackboard, Inc. have developed widely-used and helpful tools for those concerned 
about online course quality. These are compared with the AECT Instructional Design Standards for Distance 
Learning in the table below. 


Instructor Competencies: 
Standards for Face-to- 
Face, Online & Blended 
Settings 


International Board of 
Standards for Training, 
Performance and 
Instruction (ibstpi) 


Professional development 
and evaluation of 
instructors 


Instructors, academic 
leaders, human resource 
development and 
professional development 
personnel 


Instructional Designer 
Competencies: The 


International Board of 
Standards for Training, 


Professional development 
and evaluation of 


Instructional designers; 
ID managers; academic 


Standards Performance and instructional designers leaders 
Instruction (ibstpi) 
Quality Matters Quality Maryland Online/ Formative evaluation of Course evaluation 
Matters Higher Education | Quality Matters online course quality reviewers or teams; 
Course Design Rubric instructional designers, 
instructors developing 
courses 
OSCQR Course Quality Open SUNY Center for Formative evaluation of Course evaluation 
Review Rubric Teaching Excellence. online course quality reviewers or teams; 
Also distributed by instructional designers, 
Online Learning instructors developing 
Consortium (OLC) courses 
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Educators administrators 
and policymakers 


International Association 
for K-12 Online Learning 
(GINACOL) 


INACOL National 
Standards for Quality 
Online Courses 


Help constituent groups 
make informed decision 
regarding blended and 
online learning 


Summative evaluation of 
online course quality 


Blackboard Exemplary 
Course Competition 


Blackboard Exemplary 
Course 


Instructional Design Association for Provide research-based | Instructors developing 


Standards for Distance Educational guidelines for courses; instructional 
Learning Communications and developing distance designers; educational 
Technology (AECT) learning courses leaders; course 


reviewers 


The ibstpi standards emphasize competencies that can be demonstrated by instructors, instructional 
designers, evaluators, training managers and online learners (Klein, Spector, Grabowski & de la Teja, 2004; 
Kozalka, Russ-Eft, & Reiser, 2013). The Quality Matters Higher Education Course Design Rubric and the OLC 
OSCQR Course Design Review Scorecard are comprehensive tools providing a large number of assessment items 
by which the features of online and blended/hybrid courses can be evaluated formatively for improvement of 
summatively for judgment and awards (Blackboard, 2017; Maryland Online 2017; Online Learning Consortium, 
2018). The iNACOL Standards (2011) are intended for use in K-12 schools, but have much in common with the 
aforementioned standards and rubrics. 

The AECT Instructional Design Standards for Distance Learning are intended to inform and provide 
guidance before, during and after the design and development of online and blended/hybrid courses. They can be 
used in tandem with other tools to assure that empirically sound principles of learning and instruction are “baked” 
into courses designed for learners at a distance. There is also a set of accompanying sample rubrics that have been 
developed for practical application of the standards (see the next section). 


Utilizing the AECT Standards 


A set of rubrics has been developed to provide guidance for instructional designers and others who wish to 
incorporate the Instructional Design Standards for Distance Learning (Harris, 2017b). 


Standard 1: Purpose. 


Purpose statement 
is multi- 
dimensional. 


Statement incorporates 
multiple viewpoints and 
clearly articulates purpose 
as specifically applicable to 
the institution, the 
instructor, and the student. 


Statement recognizes 
multiple viewpoints and is 
generally applicable to the 
institution, the instructor, 
and the student. 


Statement is generally 
applicable but does not 
adequately address one or 
more viewpoints among the 
institution, the instructor, 
and the student. 


Purpose statement 


Statement is made through 


Statement is a generalized 


Statement is not reflective 


incorporates collaboration between the reflection of instructor and _ | of collaboration. 
collaboration. instructor and the student. student views. 

Goals and Statement includes Statement includes basic Goals and objectives are 
objectives are comprehensive elaboration | goals and objectives that missing or only partially 
articulated. through specific goals and are comprehensive and at developed. 


objectives that are coherent 
and fully articulated. 


least partially detailed. 
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Purpose is aligned 
with external 
requirements. 


Statement aligns fully with 
external requirements, such 
as state or federal standards, 
and alignment is detailed 
and specific. 


Statement generally aligns 
with external requirements 
with at least partial one-to- 
one correspondence. 


Statement either does not 
fully align with external 
requirements, or there is 
little or no evidence that 
such requirements have 
been considered. 


Standard 2: Assumptions. 


Students’ prior Students’ prior knowledge Students’ prior knowledge | Students’ prior knowledge 

knowledge is assessed in detail and is assessed in general terms | is assumed rather than 
such information is used as__| and such information is assessed. 

a primary factor to shape used to help shape course 
course design. design. 

Curricular Curricular expectations are | Curricular expectations are | Curricular expectations are 

expectations clearly articulated and generally stated and used unstated or non-specific. 
incorporated into the course | to shape the course design. 
design. 

Institutional Institutional requirements Institutional requirements Institutional requirements 

requirements are clearly articulated and are generally stated and are unstated or non-specific 
incorporated into the course | used to shape the course 
design. design. 

Technology skills Students’ ability to use Students’ ability to use Students’ ability to use 
required technology is required technology is required technology is 
assessed and such basically assessed and used | assumed rather than 
information is a factor in to help shape course assessed 
course design. design. 


Standard 3: Sequence. 


Sequence is 
consistent with 
prior knowledge. 


Students’ prior knowledge 
assessment is fully 
incorporated into the 
learning opportunities 
sequence. 


Students’ prior knowledge 
assessment is used in 
general terms to sequence 
learning opportunities. 


Students’ prior knowledge 
is not a major factor in 
determining the sequence of 
learning opportunities. 


Sequence is varied 
in accordance with 
learning needs. 


Various models of 
sequencing are chosen, 
based on the student’s 
learning needs. 


The sequencing model is 
chosen based on the 
student’s learning needs 
but is relatively static. 


The sequencing model is 
based on factors other than 
the student’s learning needs. 


Sequence 
compliments 
content. 


The sequence models are 
well matched to content for 
optimal learning. 


Sequencing generally 
compliments content. 


Sequencing is determined 
independent from content. 


Sequence optimizes 
delivery. 


Sequencing is determined in 
order to provide a best fit 
within online delivery 
constraints. 


Online delivery constraints 
are taken into 
consideration when 
choosing sequence. 


Online delivery constraints 
are not well matched to 
chosen sequence. 


Standard 4: Activities. 
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Activities are 
varied. 


Activities combine a variety 
of passive and active forms 
of engagement. 


Activities combine some 
forms of active and passive 
engagement. 


Activities tend to be mostly 
limited to either active or 
passive engagement, not 
both. 


Activities are 
matched to 
knowledge, 
experience, and 


ability. 


Activities are chosen based 
on the student’s specific 
level of knowledge, 
experience, and ability. 


Activities generally suit the 
student’s level of 
knowledge, experience, 
and ability. 


Activities tend to be 
predetermined, rather than 
specifically related to the 
student’s knowledge, 
experience, or ability. 


Activities are self- 
selected or self- 
developed by the 
student. 


Activities balance self- 
selected/self-developed 
options and instructor- 
selected/instructor- 
developed options. 


The student’s self- 
selected/self-developed 
activities are given 
consideration and included 
whenever possible. 


Instructor- 
selected/instructor- 
developed activities 
dominate, with little 
accommodation for the 
student’s self-selected/self- 
developed activities. 


Activities match 
online delivery 
constraints. 


Activities are highly 
adaptable and provide for 
synchronous, asynchronous, 
and mixed delivery. 


Standard 5: Resources. 


Resources foster 
deep learning. 


Resources offer multiple, 
rich avenues to deepen 
understanding and extend 
learning beyond course 
content. 


Online delivery constraints 
are taken into 
consideration when 
choosing activities, and 
synchronous and 
asynchronous activities are 
included whenever 
possible. 


Resources are varied and 
provide avenues to deepen 
and extend course content 
learning. 


Online delivery constraints 
do not accommodate both 
synchronous and 
asynchronous activities. 


Resources tend to be limited 
to course-centered content. 


Resources are 
multimodal. 


Resources are based on the 
student’s specific 
understandings and 
capacities of knowledge, 
experience, and ability. 


Resources generally suit 
the student’s level of 
knowledge, experience, 
and ability. 


Resources are general, 
rather than specifically 
related to the student’s 
knowledge, experience, or 
ability. 


Resources are 
consistent with 
technological 
accessibility. 


Resources fully take into 
account technological 
accessibility to ensure that 
the student can use the 
resources both within the 
course structure and 
independently. 


Resources generally 
recognize limits of 
technological accessibility 
and ensure that the student 
can fully use the resources. 


Resources do not fully take 
into account technological 
accessibility, making some 
resources difficult or 
impossible for the student to 
use. 
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Resources 
encourage self- 
directed learning. 


Resources are consistent 
with course content and 
provide avenues for the 
student to engage in self- 
directed, extended learning. 


Resources are consistent 
with course content and at 
least some offer ways the 
student can extend learning 
through self-direction. 


Resources are consistent 
with course content but may 
be difficult or impossible for 
the student to use in 
independent learning. 


Standard 6: ne 


Application is 
integral to the 
course design. 


Application offers multiple, 
rich opportunities to deepen 
understanding through 
practice of newly acquired 
skills and knowledge. 


Application provides 
varied opportunities to 
deepen and extend course 
content learning through 
practice. 


Application tends to be 
limited or isolated from 
course content. 


Application 
provides for 
collaborative and 


Application provides many 
opportunities and 
encourages the student to 


Application offers multiple 
opportunities for 
independent and 


Application is limited and 
includes few opportunities 
for either collaboration or 


outside the course 
setting. 


to interact with peers 
outside the course setting, 
using face-to-face as well as 
electronic modes of 
communication. 


Standard 7: Assessment. 


Assessment is 
formative. 


Assessment is an integral 
part of the learning 
sequence to ensure that the 
student’s acquisition of 
knowledge and skills is 
optimal. 


electronic communication. 


Assessment provides for 
logical points of feedback 
and review over the 
learning sequence. 


independent work with others and collaborative practice of self-directed learning. 
learning. independently to practice new skills and knowledge. 

new skills and knowledge. 
Application Application includes rich Application incorporates Application includes only 
includes feedback. | feedback from the instructor | instructor and peer limited feedback. 

and multiple student peers. feedback. 
Application Application is enriched Application incorporates Application includes few if 
incorporates through multiple collegial interaction, both any opportunities for 
collaboration opportunities for the student | face to face and through collegial collaboration 


outside the class setting 


Assessment is limited or 
tends to be summative 
rather than formative. 
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Assessment is 
formal, informal, 
and incidental. 


Assessment provides 
multiple opportunities for 
formal and informal review 
as well as encouraging 
incidental review whenever 
the need arises. 


Assessment incorporates 
both formal and informal 
review and allows for 
incidental review when the 
need arises. 


Assessment tends to be one- 
dimensional, either formal 
or informal rather than both. 


Assessment fosters 
review of 
operational design. 


Assessment is key to 
reviewing both the student’s 
learning and the operational 
design of the course, which 
is flexible and subject to 
adjustment. 


Assessment is used to 
review not only the 
student’s learning but also 
the operational design of 
the course. 


Assessment is limited to the 
student’s learning. 


Assessment makes 
use of student 
input. 


Assessment is largely driven 
by student input in order to 
ensure optimal learning 
through operational redesign 
of the course on an ongoing 
basis. 


Assessment incorporates 
the student’s input in the 
revision of course design 
as needed. 


Assessment is largely 
instructor-directed or 
instructor-determined. 


Standard 8: Reflection. 


Reflection is an 
integral part of the 
operational design. 


Reflection is integrated into 
the course design so that it 
occurs naturally at 
significant intervals as well 
as spontaneously when the 
need arises. 


Reflection is included at 
regular intervals in the 
course design. 


Reflection seems to be an 
after-thought, if it is 
included at all 


Reflection extends 
feedback and 
review. 


Reflection provides a 
regular means of extending 
feedback and review 
activities and contributes to 
reshaping the operational 
design. 


Reflection actively extends 
feedback and review 
activities. 


Reflection may extend the 
feedback and review 
activities but that does not 
seem to be its central 


purpose. 


Reflection includes 
both instructor and 
student self- 
reflection. 


Reflection offers multiple 

opportunities for instructor 
and student self-reflection, 
both shared and individual. 


Reflection incorporates 
opportunities for instructor 
and student self-reflection. 


Reflection, when it occurs, 
is limited. 


Reflection deepens 
learning. 


Reflection is regularly 
employed as a means of 
deepening learning at all 
stages. 


Reflection is consciously 
used to deepen significant 
learning experiences. 


Reflection only 
serendipitously deepens 
learning. 


Standard 9: Independent Learning. 
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Independent 
learning is 
incorporated into 
the operational 
design. 


Independent 
learning includes 
feedback, review, 
and reflection. 


Independent learning is as 
important in the operational 
design as structured 
learning. 


Independent learning, 
through feedback, review, 
and reflection, helps to 
direct or redirect the 
course’s operational design. 


Independent learning 
opportunities are regularly 
occurring in the 
operational design. 


Independent learning 
parallels the operational 
design in terms of 
feedback, review, and 
reflection. 


Independent learning occurs 
or is encouraged only 
serendipitously or 
occasionally. 


Independent learning is 
unstructured 


Independent 
learning is included 
in both 
synchronous and 
asynchronous 
activities. 


Independent 
learning is both 
instructor- and self- 
directed. 


Independent learning is 
incorporated in both 
synchronous and 
asynchronous activities but 
is particularly emphasized 
in asynchronous activities. 


Independent learning is 
equally valid and essential 
whether instructor- or self- 
directed. 


Independent learning is 
encouraged in both 
synchronous and 
asynchronous activities. 


Independent learning 
includes both instructor- 
and self-directed learning 
activities. 


Independent learning, if it 
occurs, tends to happen only 
during either synchronous 
or asynchronous activities 
but not both. 


Independent learning, if it 
occurs, is either instructor- 
directed or self-directed but 
not both. 


Standard 10: Evaluation. 


Evaluation is 
purpose-driven. 


Evaluation is based 
on student 
acquisition of new 
knowledge, 
understandings, 
and skills 


Evaluation is based 
on instructor self- 
evaluation. 


Evaluation is fully aligned 
with the stated purpose(s) of 
the course and based on 
multiple factors; evaluation 
is used to shape future 
iterations of the course. 


Evaluation incorporates 
multiple factors to judge the 
success of the student’s 
acquisition of new 
knowledge, understandings, 
and skills. 


Evaluation is based on the 
instructor’s self-evaluation 
as a co-equal element in the 
multidimensional evaluation 
of the course and its design. 


Evaluation is aligned with 
the course purpose(s). 


Evaluation is 
multidimensional and fully 
takes into account the 
student’s acquisition of 
new knowledge, 
understandings, and skills. 


Evaluation incorporates the 
instructor’s self-evaluation 
of the course and its 
operational design. 


Evaluation is only 
somewhat related to the 
stated purpose(s) of the 
course. 


Evaluation does not fully 
incorporate an accounting of 
the student’s acquisition of 
new knowledge, 
understandings, and skills. 


Evaluation does not include 
or only partially considers 
instructor self-evaluation. 
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Evaluation is based | Evaluation is based on the Evaluation incorporates the | Evaluation does not include 
on student self- student’s self-evaluation as | student’s self-evaluation of | or only partially considers 
evaluation. a co-equal element in the the course and its student self-evaluation 
multidimensional evaluation | operational design. 
of the course and its design. 


Conclusion 


The Jnstructional Design Standards for Distance Learning represents a unique and notable entry into the 
design and development of online learning by the leading international association for the study and practice of 
instructional design and technology. The authors actively solicit feedback regarding the experiences of those who 
use the standards and rubrics and who have recommendations for their utilization and improvement. Feedback can 
be sent to AECT@aect.org. 
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Descriptors: Systems Thinking, Instructional Design 


Introduction 


Systems theory spans many disciplines and several decades. The field of biology has informed systems 
theory, according to von Bertalanffy (1972) and Boulding (1956), as an organizing principle of living beings. 
Extensions of this theoretical framework have led to the application of systems theory as systems thinking in a 
diverse array of organizational fields, including but not limited to health care, economics, and engineering. Within 
each field, different terminologies have emerged to describe distinctive environments with different contextual 
situations. 

Across numerous organizational disciplines, systems thinking is based on two coherent focal elements: (1) 
a single, compelling, overarching goal or purpose for the system and (2) connectivity among parts of the system, 
including departments, processes, and people, to bring forth that single goal or purpose (Checkland, 2000). The 
organizational discipline we focused on in this session was engineering. 

Areas ranging from public health to homeland security, to workforce development are enriched by applying 
an engineering perspective of complex social systems. Thomas Gilbert’s (1996) Human Competence: Engineering 
Worthy Performance provided a valuable perspective for examining how organizations as systems can refine 
themselves for the purpose of performing according to their respective visions. Gilbert usefully distinguished 
between the world views of scientists and engineers. Scientists, he noted, examine the world to determine how it is. 
In contrast, engineers are intent on remaking the world through active design (Gilbert, 1996). 

The study of engineering has provided a foundation for exploring how complex social systems facilitate the 
productive functioning and development of organizational capacity. Researchers in engineering have compiled, 
defined, and clarified the systems principles for this field (Adams, Hester, Bradley, Meyers, & Keating, 2014; 
Keating, Peterson, & Rabadi, 2003; Whitney, Bradley, Baugh, & Chesterman, 2015). Adams et al. (2014) proposed 
a construct for systems theory that specified principles applied through axioms. Their intent was to support systems 
theory by making it more understandable, meaningful, and pragmatic. It was their view that the construct of axioms 
and systems principles would “provide improved explanatory power and predictive ability,” offering context to 
“enable thinking, decision, action, and interpretation with respect to systems” (Adams et al., 2014, p. 120). 

Systems thinking as understood in the field of engineering also has relevance for instructional designers. 
Consistent with the engineering philosophy of complex organizational systems, instructional designers craft 
interventions designed to facilitate an organization’s functioning at a high level of effectiveness. Understanding 
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organizational systems provides a logical starting point for such work. Clarity is of importance at every phase of 
instructional design, with the purpose of building system capacity to increase the probability of achieving the desired 
outcomes. 

As part of our work as consultants, we often design and deliver instruction for professional development, 
and we view complex social systems as a fundamental reality. Such complex organizational systems typically exist 
to focus on large-in-scope goals that transcend what any single organization can do alone (Jones, 2014). Strategic 
thinking and planning also play vital roles in designing complex organizational systems to achieve shared success. 
Such work within complex systems requires education on systems thinking as a vehicle for making productive 
change. 


Systems Principles and the Instructional Design of Professional Development 


How can professional development become more deeply relevant and meaningful? Systems thinking 
provides a foundation for building the workplace as an entity that enriches and educates. Designing professional 
development from a foundation grounded in organizational systems principles enhances the designer’s 
understanding of how the system works and the degree to which professional development activities and approaches 
can be effective. 

An examination of the nature of organizations as systems strengthens the designer’s understanding of the 
context framing professional development. Recognizing the characteristics of a system informs the understanding of 
the sphere within which professional development takes place. An in-depth examination of the organizational system 
through the lens of specific principles results in a clearer understanding of how the system works (Adams et al., 
2014). Such study further clarifies the degree to which an instructional design can be effective. 

During this session, three systems thinking principles were introduced, defined, and discussed in the broad 
context of designing and participating in professional development. Examples from the private sector, public sector, 
and higher education were explored. Participants engaged in activities to guide practice in the application of systems 
principles. 

This session presented these principles from the perspective of designing professional development. The 
learning outcomes for this session were: 

e Identify accurate, authentic, and realistic organizational performance by applying the Law of 

Consequent Production. 

e Examine the systems principle of Complementarity as it pertains to learners, the organization, and 

continuing professional development. 

e Propose the systems principle of Holism as a lens through which to view the field of instructional 

design. 


Theory to Practice 
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Figure 1. Theory to Practice 
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Theory to Practice 

As noted earlier, systems theory provides a solid foundation from which we can trust the work we do as 
instructional designers. As shown in Figure 1, to translate systems theory into practice, we first follow a deductive 
process. Beginning with systems theory, we discern which systems principles are most relevant for instructional 
designers when designing professional development. After singling out salient and relevant aspects of the theory, in 
this case three systems principles, we move to the design stage by asking questions. As we reason, discuss, and 
refine, a design begins to take shape for delivery to participants. During delivery, participants are encouraged to 
provide their perspectives regarding the application of systems principles. In particular, participants may identify the 
level of effectiveness in applying a particular systems principle. Through an inductive process, the resulting 
interpretations stimulate the recognition of a newly discovered theoretical perspective. The inductive process 
supports the delivery as well as the application. In this way, theory informs practice and practice informs theory. 


Systems Principles and Instructional Design Practices 

The three focal systems principles provide a rich foundation for performing instructional design from a 
systems perspective. As shown in Table 1, these principles correlate to three key practices inherent in instructional 
design. 


Table 1. Systems Principles and Instructional Design Practices 


Systems Principle Law of Consequent Production Complementarity Holism 


Instructional 


; ; Needs Assessment Learner Analysis Contextual Analysis 
Design Practice 


The Law of Consequent Production 

The first principle discussed was the Law of Consequent Production, which states that a system can only 
produce what it produces (Keating et al., 2003). This principle is associated with a needs assessment conducted in 
concert with the instructional design of professional development programs. Needs assessment facilitates identifying 
a gap between ‘what is’ and what ‘should be’ in an organization (Altschuld & Kumar, 2010). The Law of 
Consequent Production examines what a system is producing as the best indicator of how a system is designed. The 
introduction of this principle may provide additional theoretical foundation to the application of needs assessment by 
identifying accurate, authentic, and realistic organizational performance. 

The Law of Consequent Production helps designers discern between the purported intention and the actual 
production of a system or project. Many times, we are tempted to say, “Here is what we would like the system to 
produce.” Consider, for example, a K-12 district in which the school board, the administration, the faculty, the 
parents, and the students, regard their schools as excellent. Now suppose a concerning reality emerged: the 
graduation rate at the high school is only 60%. 

According to the Law of Consequent Production, what the system produces is the best indicator of how the 
system performs. This law anchors the fundamental practice of needs assessment. In performing the needs 
assessment, we confront the gap between what is claimed and what exists in reality. 

During this session, participants worked in groups and selected an example of a system from those 
provided to support learning about the Law of Consequent Production. Participants from the groups indicated the 
purported design of their system and what that system actually produced. 

When conducting a needs assessment, instructional designers often confront a contrast between what they 
would like to be seeing happening from the system, and the actual result. It seems fair to suggest that the gap may 
exist due to a breakdown in communication. Perhaps some of the stakeholders were not part of the dialogue at the 
front end of the needs assessment as to what was being configured for the system. 

With this systems principle, we are reminded that if what the system produces is not what is intended, we 
may benefit from reverse engineering. For example, as instructional designers conducting the needs assessment, we 
might state, “Let’s find out how we produced that undesirable result. Let’s work backwards to identify the processes 
involved.” As we do that, we can recalibrate how we arrived at that result. Then we can adjust or redesign for the 
desired result. The Law of Consequent Production introduces a reality check as we perform a needs assessment. This 
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systems principle acknowledges the truth of the ‘what is’ reality, enhancing the thinking of an effective needs 
assessment. 


Complementarity 

The second systems principle was Complementarity, defined as: Each person’s perspective is both valid and 
incomplete (Adams et al., 2014). This extremely powerful systems principle invites us to gather a vast array of 
perspectives in the interest of accuracy and clarity. 

Participants in this session engaged in an activity to identify the variety of perceptions possible about 
several images of organizational life provided by the presenter. The discussion emphasized what makes each 
perspective valid and how the range of perspectives contributes to the accuracy of the picture. Specific questions 
included: What if we considered every perspective as valid — even and especially those we disagree with? What if 
we considered those difficult valid perspectives as constraints within which we must design the organization, 
department, instruction, etc.? What insight might this principle provide? 

This systems principle informs the performance of learner analysis by instructional designers (Brown & 
Green, 2016). Learner analysis as informed by the systems principle of complementarity benefits instructional 
design in a systems-enhancing way, anchoring a level of accuracy that depends on gaining as many views as 
possible, arriving at a clarity of system reality from multiple views, at once valid and incomplete. 

In the current societal environment, we are challenged by many viewpoints. As we examine the many 
views, their simultaneous validity and insufficiency, civility is critical. The more we hear, the more we learn, and the 
closer we arrive at an elevated understanding. The instructional designer needs to recognize a compendium of 
viewpoints to discern priorities and the key message for that system. This means slow, deliberate listening and 
synthesis. 

The listening and synthesis of insights allows the simultaneous validity and insufficiency of each 
perspective, leading to improved clarity about the system reality. When we perform learner analysis, we’re 
discovering factors that enrich our understanding of what the system needs to do. This means we are working 
deductively, reasoning how the system is defined from the many viewpoints, all valid and incomplete. The infinity 
of views fortifies the sharpening of a clear and accurate understanding of the system. Performing this learner 
analysis as it is informed by the systems principle of complementarily means an enhanced approach to instructional 
design based on a systems reality. 


Holism 

The third and final systems principle examined was that of holism. Holism is defined as perceiving, 
understanding, and appreciating a system from its collective essence (Whitney et al., 2015). 

During the session, the presenter displayed a simple diagram of a university system on the board, 
represented by dots (departments) and lines (relationships). The presenter proposed the joining of two unlikely areas 
of the organization and requested that participants identify what might be possible if those connections were created, 
interrupted, or elaborated on. 

Holism informs contextual analysis (Adams & Keating, 2011) required of instructional designers to ensure 
an accuracy of purpose and fit for the initiative or product. Recognizing the parts of a system as inherently and 
practically connected, it is important to understand that effective performance often includes different people and 
departments contributing particular activities and resources to complete the work. At the system level, processes 
themselves are interconnected, as well. Thus, the holistic perspective informs an overarching emphasis on how each 
contribution is relevant and important and serves the system purpose or goal. 

Emphasis is placed on the connections between parts of the system, noting that these connections are 
simply relationships that are created or invented rather than pre-existing. They can be made stronger, weaker, 
elaborated on, or erased. They can connect to the vision of the organization, the heart and mind of the individual, to 
the expansiveness of the universe as a system, or anywhere in between. Thinking holistically means thinking in 
more than two dimensions. Such thinking adds depth, richness, innovation, and inspiration. This practice allows for 
and encourages an understanding of the social nature of humanity as well as opportunities for technological 
advances. 

As instructional designers performing contextual analysis, we need to know where and how the learner is 
situated. Is there managerial support for that person in his or her job? What other supports are present within the job 
environment? This means also that we need to be alert to the non-linear and inclusive pieces that allow for an 
emergence of aspects we might not understand or anticipate regarding the context of the system-level performance 
and results. 
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The principle of holism emphasizes connections among parts of the system while maintaining clarity about 
the purpose and direction of the complex organizational system. Holism reflects a palpable and fundamental reality 
of the big picture of a system, revealing the depth, richness, innovation, and inspiration potentially available. Holism 
also provides a contextual lens through which to view the field of instructional design. 


Conclusion 


Meaningful change management initiatives benefit from being situated in the framework of systems 
thinking (Gharajedaghi, 2011). The strong and multiple connections enhance the capacity of the complex 
organizational system to produce results through systems thinking. Examining systems principles enriches the 
understanding of instructional design practices that facilitate the effective design of professional development 
initiatives. 

Three specific systems principles were examined as they inform three parallel instructional design 
practices. The systems principles of the law of consequent production, complementarity, and holism were presented 
as parallel to the instructional design practices of needs assessment, learner analysis, and contextual analysis. 

The law of consequent production informs what we know as needs assessment in our instructional design 
work. Complementarily informs learner analysis and positions us to discover yet more perspectives. Holism is what 
informs contextual analysis. As we connect these three systems principles, we discover that they work together 
resiliently and informingly so that what we do in instructional design provides a deeper foundation and connection 
among them. 

As we view holism in relationship to complementarity and the Law of Consequent Production, we 
recognize an integrated whole that is fortified by an array of diverse views that could address gaps in a system’s 
productivity and enhance the larger picture of the organization. Together, the three systems principles support the 
two primary features of an organizational system: (1) A single unified goal for the entire complex system, and (2) 
The inter-relationships among parts of that system. 

Complex organizational systems evolve, change, and strengthen. The three systems principles described 
and explored in this session inform each of the respective instructional design practices to produce value for the 
professionals within these systems. In other words, strengthening analytical practices with systems thinking serves 
to enrich the instructional design of professional development. 
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Abstract 


The Magis Instructional Design Model endeavors to transform teaching online through the lens of critical pedagogy 
to place the human in a real-world context as much as possible through learning experiences and reflection. The goal 
being transformative learning experiences instead of transmissive ones that use the antiquated banking model of 
education. The model includes instructional strategies from the cognitive and affective domains. The Author asks for 
input and feedback on this model. 


Introduction 


As an instructional designer, I seek to provide course design guidance that is inclusive and just to overcome 
environmental barriers through a humanistic approach. The Magis Instructional Design Model endeavors to 
transform teaching online through the lens of critical pedagogy (e.g., Ignatian pedagogy, Freirean praxis, justice 
education) to place the human in a real-world context as much as possible through learning experiences and 
reflection. The goal being transformative learning experiences instead of transmissive ones that use the antiquated 
banking model of education where the teacher deposits knowledge for the student to withdraw (Bradshaw, 2017). 
The design for transformative distance education (DE) includes (a) instructional strategies from the cognitive and 
affective domains, (b) planned interactions for cognitive, social, and teaching presences to engender a community of 
inquiry (COD), and (c) the universal design for learning (UDL) to meet the needs of all learners. 

This model is called Magis because it means more or greater, which refers to Saint Ignatius’ recognition 
that the deepest reality of the world, and of each individual, flows out of the infinite mystery of God (Traub, 2010). 
Magis is used in connection with another Latin term, ad majorem Dei gloriam, which means ‘to the greater glory of 
God’ (Traub, 2010). In this sense, magis means doing more for God’s project for the world and each person. 
Therefore, Magis is a fitting name for this ID model that embeds the transformational ideas of Saint Ignatius (i.e., 
Ignatian pedagogy) in concert with other critical pedagogies. 

Ignatian pedagogy focuses on teaching to the whole person through context, experience, reflection, action, 
and evaluation (Kolvenbach, 1987; Korth, 1993). These principles are based on the work of Ignatius’s Spiritual 
Exercises, which serve as a guide to enhance one’s vocational discernment (Modras, 2008). Albeit steeped in 
religious overtones, the Magis model can be used for secular ID purposes. For example, Jesuits focus on social 
justice issues and aid the marginalized in their teachings as part of the Society of Jesus. Critical pedagogies also 
share this mantra but in a secular approach. 


Model Overview 


An ID model is an iterative process that analyzes and then synthesizes the variables affecting the learner, 
learning context, content to be taught, & application of the information (Dick & Carey, 2009). The purpose of this 
model is to align instruction with critical pedagogy for the adoption of transformative (democratic) teaching and 
learning to avoid transmissive (authoritarian) education. It is outcome-referenced and conditions-based. The 
outcomes address the Jesuit and Catholic mission to transform learners into caring leaders who seek truth, appreciate 
the beauty of life and God’s love, and promote human solidarity (Fr. Lucey, 2015). This formation is lifelong, as the 
distal outcomes of a Jesuit-based education. The conditions include the cura personalis of Jesuit education to engage 
students holistically, through the formation of their mind, body, and spirit. These are the conditions for Ignatian 
pedagogy and, to a broader extent, the proximal outcomes of transformational teaching to meet the needs of all 
learners. 

Rationale. The ID process can reflect ethical considerations as input (Moore & Ellsworth, 2014). 
Educating the whole person is an ethical issue. Recognizing students as humans is something we intuitively do in 
traditional face-to-face classrooms, but sometimes this is lost in DE. For example, if e-learning is only a text-based 
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independent study, then we’ve silenced our students and treated them as mutes. The U.S. Office of Education (2010) 
conducted a meta-analysis of 99 e-learning studies and found larger effect sizes for studies that included 
collaborative or teacher-directed learning activities than those with independent study. Thus, independent online 
coursework is not always as effective as interactive online courses. The alternative, a transmissive education of 
information dumping, is unethical. Freire (1970) called it a corpse of knowledge. 

Utility. The Magis model is designed for hybrid and fully online DE. It is a micro-level model that 
integrates critical pedagogy through a systemic approach. Micro-level models address instruction at the course level 
(not the program level) and are considered an adaptive instructional format (Lee & Park, 2008). Adaptive instruction 
provides differentiated learning per the learners’ needs and abilities. It does not need to be technology-based. 
Adaption at the macro-level allows for a variance with instructional goals, depth of curriculum, and type of delivery. 
The Author decided to create a micro-level model instead of a macro-level programmatic one because of the spirit of 
creativity and individuality that educators have. 

Uniqueness. This Magis model is unique in that it includes religion, spirituality, and social justice in 
addition to intellectual growth. It is inclusive of service to others. According to Dr. Timothy Carmody at Spring Hill 
College, Jesuit educators focus instructional activities on experiential learning to engender the cycle of experience 
leading to reflection and further action. This is based on the dynamics of Ignatius’ Spiritual Exercises. 

Limitations. Initially, this model was designed for DE at Jesuit and Catholic institutions. In reflection, the 
Author realized the paradigm in which it belonged---critical pedagogies for all types of online educational programs. 
The challenge is a more inclusive model of the various critical pedagogies instead of the dominant Ignatian one. 
More research and practice are needed in addressing transformational teaching practices in DE. 


Theoretical Basis 


The Magis model is based on a multi-theoretical approach to the psychology of learning according to these 
paradigms: behaviorism, cognitivism, and constructivism. Learning is the acquisition of knowledge, skills, abilities, 
as well as the acculturation of values, attitudes, and emotional reactions (mindset). Regarding behaviorism, learning 
is determined from observations such as the completion of a new behavior/task, change in frequency/speed/intensity 
to said task, change in the complexity of a task, and responding differently to a particular stimulus (Ormrod, 2012). 
As for cognitivism, learning can be inferred from certain situations like avoidance of risky or unpleasant behaviors; 
it does not need to be explicitly observed (Ormrod, 2012). As for constructivism, learning is socially constructed by 
a student’s observation of others and co-constructed with their interactions with others (Ormrod, 2012). The Author 
considers constructivism to be a subset of cognitivism. 


Conditions of Learning 

The Magis model is influenced by Gagné’s (1985) work on the conditions of learning (internal and 
external), domains of learning (1.e., attitudinal, cognitive strategies, intellectual, psychomotor, or verbal), and the 
instructional approach known as the nine events of instruction. His work is based on a multi-theoretical learning 
paradigm. Conditions-based instructional theories address how and when to provide feedback, motivation, mastery 
learning, instructional sequencing, and learner strategies. Gagné, Briggs, and Wager (1992) worked to further define 
intellectual skills into a classification system in the following ascending order of difficulty: discrimination, tangible 
concepts, defined concepts, rule usage, and higher-order rules. This classification system is useful for the Magis 
model to design critical thinking interactions to empower students. 


Zone of Proximal Development (ZPD) 

Vygotsky’s (1978) proposed that learning takes place at the edge of one’s understanding with the help of 
others or a support system. This is known as the ZPD. This means that learning will not take place if the activity is 
too easy or too difficult. Csikszentmihalyi (1990) also described flow occurring for activities within a channel with 
just the right type of challenge to match a person’s skills. This channel exists somewhere between anxiety and 
boredom. Educators understand the need for differentiated instruction to meet each learner’s needs, but the reality of 
trying to make this happen in a traditional classroom of diverse learners is almost impossible to do all the time. This 
is where DE has an advantage over face-to-face instruction, as it provides more flexibility in content delivery, 
affordances for accommodations, modifications, and groupings. Hybrid formats can provide the best of both types of 
instructional environments. 
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Community of Inquiry 

The Magis model incorporates instructional practices, instructional design, and technologies that support an 
online COI. The essential elements of a communication loop for an online COI include cognitive, social, and 
teaching presences (Garrison, Anderson, & Archer, 2000). This means that learners in an online environment are 
involved in cognitively challenging activities, can interact with their classmates, and that the teacher or student 
moderator is present in some way through words, voice, or person. 

We need to bring the learner, the content, and the instructor together in an online community of inquiry for 
DE. Rogers and Van Haneghan (2016) created the Online Community of Inquiry Syllabus Rubric© with criteria for 
consideration of an exemplary online COI. The rubric includes criteria for cognitive presence, technology tools for a 
COI, COI loop for social presence, support for learner characteristics, and instruction and feedback for teaching 
presence. The function of the rubric is to review the potentiality of an OCOI based on the course syllabus in the 
developmental phase. Its purpose is similar to Palmer and Caputo’s (2003) Universal Instructional Design model 
that has an implementation guide for instructors to reflect on the course structure and activities. Since the syllabus 
serves as a plan of action, this rubric serves as a tool during the analysis phase of the ID process to determine the 
strengths and weaknesses of an instructor’s plans for online ITs. The underlying theoretical premise is the more 
interactive and cognitively challenging the course, the higher the level of student satisfaction and course 
achievement. See Appendix A for the rubric. 

Designing a COI loop of planned interactions will address teaching to the whole person. Bernard et al. 
(2009) conducted a meta-analysis of 74 online course interactions and found substantive research outcomes 
indicating the positive effect on learning when online educators build these types of interactions into their courses: 
student-student, student-teacher, and student-content. These interaction treatments were defined as the environments 
and not the actual behaviors that occur within them. Through the ID processes, one can design and develop these 
types of settings for DE. Table | displays the interrelationship of the main components of the model. Each 
component aligns to some extent with the mind, body, and spirit of curas personalis---the care of the entire person. 


Table 1. Components of the Magis Instructional Design Model 


Jesuit Education Mind Body Spirit 
(Korth, 1993) 


Community of Inquiry Cognitive Presence Social Presence Teaching Presence 
(Garrison, Anderson, & (includes student-led 
Archer, 2000) moderation of tasks or 
peer evaluation) 
Evidence-Based Student-content Student-student Student-teacher 
Educational Research interactions interactions interactions 


(Bernard et al., 2009) 


Critical Pedagogy Active, authentic, & Cooperative & Humanistic “locations of 
(Hooks, 1994) intentional constructive ethical possibility” 
engagement 


One of the non-educational influences for this model is Csikszentmihalyi’s (1990) flow theory. For the 
Magis model, the Author is interested in how it motivates one to a higher level of performance. Flow theory has 
eight main components that engender enjoyment: manageable tasks, deep concentration, clear goals, immediate 
feedback, effortless involvement, learner autonomy, the metamorphosis of self, and suspension of time 
(Csikszentmihalyi, 1990). Aspects of flow theory relate to the vocational discernment that Jesuit educators want 
their students to contemplate to achieve an elevated state in the work that they do in service to others. Designing 
instruction to engender flow can provide a more fulfilling learning experience, which is the impetus for the Author’s 
model. 


Main Principles 
The Magis model is a combination of learner-centered, experience-centered, activity-centered, and content- 
centered to address the whole person in online courses and for the UDL. Ragan, Smith, and Curda (2008) stated that 


a combination ID model is possible. Not only is it possible to include research-based best practices, but it is also 
necessary to provide diverse and vibrant experiences in online environments. Otherwise, a single-mode of learning 
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will become monotonous and decrease student motivation to learn. The UDL promotes multiple means of action, 
engagement, and representation to address the needs of all learners (CAST, 2011). 


The Magis model has iterative ID phases of analysis, design, development, implementation, and evaluation. 


Each phase (re)occurs at any time, as in concurrent design approaches. For example, the evaluation phase includes 
both formative and summative evaluations. Formative evaluations arise at every stage and include feedback obtained 
from key players, which informs the subsequent design and assessment decisions (Dick & Carey, 2009). Formative 
assessments include the use of prototypes (paper, web, or conceptually-based) and single or group feedback. 
Furthermore, concurrent design relies on constant interaction between the instructional designer, course developer, 
and instructor (Davidson-Shivers & Rasmussen, 2006). Table 2 displays the inputs and outputs of the Magis model. 


Table 2. Phases of the Magis Instructional Design Model 


ID Phases 


Procedures 


Actions 


Outcomes 


Analysis 


Analyze learner, learning 
environment, and content, as 
well as the instructors’ ability 
to teach in an online format 
with the current learning 
management system. 


Conduct a needs assessment of instructor 
skills and attitudes toward online 
environments and technology. Review 
existing syllabus, course goals, and 
objectives with the Online Community of 
Inquiry Syllabus Rubric©. Observe classes 
if possible. Obtain student input about their 
knowledge, skills, abilities, and other 
interests (KSAOs). 


Instructional designer 
addresses the needs of the 
instructor and students by 
tapping into their prior 
KSAOs. Instructional 
designer identifies areas to 
maximize the potential to 
engender an online COI to 
establish relationships of 
mutual respect. 


Design Develop inclusive course Review existing assessments and material; Course designers optimize 
goals and unit objectives. create measurable objectives at the unit the course to address the 
Create assessments for level; design rubrics, tests, and alternative whole learner. Instructor will 
content that align with course | assessments based on criteria. Scaffold be able to measure students’ 
goals and student learning cognitively challenging assignments for learning. 
outcomes. Select wholistic formative assessments and feedback loop. Students can achieve course 
instructional strategies to Select action verbs to engage higher-order goals with aligned content 
match objectives. thinking skills for assignments. and appropriate tasks. 
Develop- | Develop an online course Add content, threaded discussions, Course developers complete 
ment with participatory practices. assignments, quizzes, resources, related a beta version of course for 
Embed useful learner images; Develop multimedia presentations. | quality review. Students will 
strategies into the content to | Meet with a librarian to discuss be able to not only be able to 
include questioning and bibliographic instruction for your online learn but reflect on their 
reflection techniques. students and useful resources. Use Gagné’s | learning and challenge the 
nine events of instruction when planning status quo. 
lessons. Select online cognitive activities to 
teach the whole person and engender an 
OCOI. 
Implemen | Pilot-test course, attend to Fake students (colleagues) are added to The course is well-prepared. 
-tation issues, then add students. pilot-test course. Create course Students know course 
announcements, send introductory class expectations. Students 
email, establish online office hours, and assimilate new information 
develop a matrix of students to monitor through active learning. 
interactions. Provide list of student and 
teacher expectations for e-learning to 
promote course interactions. Poll students 
on prior knowledge of course topic. 
Evalua- Address quality assurance Informal quality assurance review of online | Student achievement and 
tion measures for online course; survey students on course design; satisfaction increase. 


instruction and accessibility 


address immediate issues and develop list of 
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needs at every phase in the course redesign items. This is part of a Students successfully apply 
ID process. continuous improvement cycle to address inquiry skills in real world 
students’ needs. contexts based on classroom 
participatory practices. 


The model’s instructional design begins with the analysis phase of the human learning experience that 
includes student input. The design and development phases are steeped in the COI and UDL frameworks and 
participatory practices of critical pedagogies. For example, through questioning and reflections beyond self-interest, 
learners can become contemplatives in action. The implementation of the course includes introductory activities to 
tap into the learners’ prior knowledge (including misunderstandings) and experience. Prior knowledge along with 
learners’ belief systems and environmental barriers affect learning (Ormrod, 2012; Bransford, Brown & Cocking, 
1999). Environmental barriers to learning include economic, physical, political, linguistic, ethnocultural, and social 
(e.g., gender bias). These barriers are important issues for social justice. Figure | relies heavily on Ignatian 
pedagogical layers to develop learners into caring leaders. It serves as the framework for the Magis model. 


Establish Relationships of Mutual Respect Online/Offline 


Figure 1. Magis Instructional Design Model Framework for Distance Education 


Instructional Strategy Selection 

Table 3 provides instructional strategies for DE that engender higher-order thinking for four different 
approaches to provide cognitive presence. This chart serves as a job aid for strategy selection during the design 
phase. This selection depends on various affordances and constraints such as time and resources. For example, an 
activity-centered lesson is based on an interactive task and requires collaborative tools and student groupings. 
Content-centered lessons are passive tasks where the student generally only interacts with the content---the 
exception being discussions of content. Experience-centered activities require a hands-on approach to developing 
something or serving/working with others. The learner-centered activity provides the learner with more autonomy 
over their pursuit of knowledge and includes metacognitive actions for self-regulation of learning; the affordances 
and constraints for this type of activity are highly dependent on the task. Providing a myriad of instructional 
strategies will lead to transformational teaching through inclusiveness and participatory practices. 
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Table 3. Cognitive Online Instructional Strategies to Teach to the Whole Person 


Activity-Centered 


Content-Centered 


Experience-Centered 


Learner-Centered 


Analysis of case studies 
Critically review an 
article 

HyperInquiry team 
project 

Academic controversy 
assignment 

Develop a book trailer on 
topic 

WebQuest 


Pretest/Posttest 

Write a literature review 
Complete modules on a 
topic in computer-adapted 
lab/program 

Write an essay 

Make a presentation 
Discuss content with 
peers and instructor 


Develop questionnaires 
Develop a personal model 
of topic 

Participate in a simulation 
Develop a workshop 
Develop a wiki on a topic 
Develop a podcast or 
narrated PowerPoint on a 
topic 

Develop a how-to guide 
on procedure 

Write a blog post on a 
topic 

Serve others as a mentor, 
tutor, or volunteer on 
topic 

Virtual field trip 


Peer-review of papers or 
projects 

Students create m/c 
questions for review 
Design a project 
Evaluate a program 
Write an autobiography of 
your interaction with 
topic 

Complete self-evaluation 
Develop a personal 
learning network 
Capture reflections in 
journal, audio, or video 
Curate digital books and 
articles on topic for 
lifelong learning 


Influential Instructional Design Models 


First-Principles. Gagné’s (1985) conditions-based theory was based on the need to align the various types 
of learning with instructional events, as were Merrill’s (2009) First-Principles of instruction. Merrill’s principles 
included the following cyclical instructional phases: task-centered, activation, demonstration, application, and 
integration. At its core was the task-centered approach based on a realistic problem. This aligns with the Ignatian 


pedagogical principles of context, experience, and action but does not include reflection. 


4C/ID. Van Merriénboer and de Croock’s (2002) Four-Component ID (4C/ID) model was for the micro- 
level design of learning of complex tasks. It was a holistic approach, like Ignatian pedagogy, that used authentic 


tasks to learn skills and knowledge in an integrated approach instead of piecemeal. The four components consisted 
of several steps: learning tasks (i.e., design and sequence tasks, set performance objectives), supportive information 
(i.e., design support information, analyze cognitive strategies and mental models), procedural information (i.e., 
design procedures, analyze cognitive rules and prerequisite knowledge), and part-task practice (i.e., design part-task 
practice). The task-specific focus was organized by level of performance difficulty. This model addressed learners’ 
issues before they become frustrated, which is an essential element in teaching to the whole person. Ignatian 
pedagogy strives to teach to the whole person. 

UID. Palmer and Caputo (2003) proposed the following seven principles for universal instructional design 
(UID): accessibility, consistency, explicitness, flexibility, accommodating learning spaces, minimization of effort, 
and supportive learning environments. Universal design (UD) refers to the consideration of the needs of persons 
with disabilities regarding physical spaces and objects. UID recognized those needs for course design. Its premise 
was equal access to education and extends this to all types of learners and not just those with disabilities. The UID 
model addressed the needs of the whole learner akin to Ignatian pedagogy. It included an explicit implementation 
guide for educators to reflect on the current course situation and act on their identification of course weaknesses 
regarding UID. Reflection and evaluation are essential principles for critical pedagogies. 


Summary 


The Author is grateful to the Jesuits at Spring Hill College for their input. Additionally, my 
acknowledgment of my instructor, Gayle V. Davidson-Shivers, Ph.D., for her support and feedback in the 
development of this model. In summary, the user of the Magis ID model will analyze human learning experiences 
during the analysis phase. During the design and development phase, users will consider the parameters of mutual 
respect in an online learning environment between T-S, S-S, and S-C in the context of humility, openness, positive 
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attitude, and reverence. Part of that respect includes tapping into the students’ prior knowledge and experience 
through questioning during the implementation phase. Users will design for the assimilation of new information for 
complete understanding using activities that engender higher-order thinking skills and the transfer of learning into 
everyday interactions. Additionally, it asks the user to include instructional components for lifelong habits and 
reflections beyond self-interest. The framework is inclusive of the mind, body, spirit in terms of a COI and other 
transformational research-based practices for the implementation of a more humanistic online education. 
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Directions: Award points to determine the course’s potential of developing an online community of inquiry (COl) 


Online Community of Inquiry Syllabus Rubric 
©2018, Sandra A. Rogers & James Van Haneghan 


through these interaction treatments identified on the syllabus: student-student (S-S), student-teacher (S-T), 
student-practitioner/expert (S-PE), and student-content (S-C). 


Low potential for building an online COI 1-9 points 
Moderate potential for building an online COI | 10-17 points 
High potential for building an online COl 18-25 points 


Scale 


Exemplary (5 points) 


Above Average (4 points) 


Moderate (3 points) 


Basic (2 points) 


Low (1 point) 


Instructional 
Design for 
Cognitive 
Presence 


Educational 
Technology 
for COl 


extensive cognitive 
activities such as 
exploration, integration, 
resolution, & triggering 


events (analysis, synthesis, 


or evaluation). 


Technology could 


extensively facilitate a COI 


(e.g., email, assignment, 


forum, multimedia project, 


sharing tool, & 


synchronous meeting tool 


for group work). 


Instructional design offers 


Instructional design offers ample 
cognitive activities such as 
exploration, integration, and 
resolution (applying new ideas), 
This is at the application level of 
inquiry. 


Technology could amply 
facilitates a COl (e.g., email, 
assignment, forums, multimedia 
project). Project sharing tool is 
used to obtain peer feedback or 
group collaboration. 


ideas). This is at the 


inquiry. 


Technology could 


Instructional design offers 
adequate cognitive activities 
such as exploration and 
integration (connecting 


comprehension level of 


adequately facilitates a COI 
(e.g., email, assignment tool, 
a forum tool). Multimedia is 
used for individual project 
for teacher’s view only. 


minimum cognitive 
activities such as 


ideas). This is at the 


Technology could 
mi 
with T-S and S-S 


tool). 


Instructional design offers 


exploration (exchange of 


knowledge level of inquiry. 


imally facilitate a COI 


interactions (e.g., email, 
assignment tool, & a forum 


Instructional design 
offers limited 
cognitive activities 
(e.g., no exchange of 
ideas) for interaction 
treatments. 


Limited technology 
provided to facilitate 
a COI. For example, 
email and/or 
assignment tool for T- 
S$ interactions only. 


COI Loop for 
Social Presence 


Support for 
Learner 
Characteristics 


Instruction & 
Feedback 

for Teaching 
Presence 


Open communication 
actions provide for 


extensive S-T, S-S, & S-P/E 


interactions and 


opportunities for student- 
led moderation of forums. 


Collaboration is required 
to build group cohesion 
and a rubric is provided. 


Extensive learner support 
and available resources are 


identified (e.g., disability 


services, remedial services, 


strategies/tips, & 


scaffolding assignments). 


Extensive information 
provided on instructor 
feedback format with 


prompt turnaround time. 


Multi-modal direct 


instruction is mentioned. 


Instructor offers virtual 
office hours, format, & 


social media for classroom 


interactions. 


Open communication actions 
provide for ample S-T and S-S 
interactions and opportunities for 
student-led moderation of 
forums, Collaboration is required 
to build group cohesion and a 
rubric or guidelines are provided. 


actions provide for 


point-system, or by 
example. 


Open communication 


adequate S-T and S-S 
interactions to discuss 
content. Collaboration is 
encouraged to build group 
cohesion through words, a 


actions provide for 


forum for 


Ample learner support and 
available resources are identified 
and offered (e.g., disability 
services, remedial services, 
strategies/tips). 


or strategies). 


Adequate learner support 
and available resources are 
identified (e.g., disability 
services & remedial services 


Open communication 


minimum S-T and S-S 
interactions such as a 


questions/answers and/or 
watercooler socializing. 


Minimum learner support 
and available resources are 
identified (e.g., disability 
services or remedial 
services or strategies). 


Communication 
actions are limited to 
S-T interactions only 
such as email. No 
open communication 
planned. 


Learner support and 
available resources 
are not fully shared 
(e.g., no contact 
information). 


Ample information provided on 
instructor feedback format with 
prompt turnaround time. Multi- 
modal direct instruction is 
mentioned (e.g., narrated slides, 
video tutorial, or digital 
program). Instructor offers virtual 
office hours and format. 


direct instruction is 


for blended course). 


virtual office hours. 


Adequate information 
provided on instructor 
feedback format. Text-based 


mentioned (or live lecture 


Instructor offers specific 


Instructor offers 


nonspecific virtual office 


hours. 


Minimum information 
provided on instructor 
feedback format. No direct 
instruction mentioned. 


Limited information 
provided on 
instructor feedback 
format. No direct 
instruction (focusing 
discussion) 
mentioned. Instructor 
does not offer virtual 
office hours. 


Notes: 
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Abstract 


In reviewing nearly 700 self-described studies of educational data mining (EDM) over the past decade, 
only 13 reviews were attempted that synthesize research literature. Within these 13, several key issues present 
themselves including non-systematic selection and the confounding of results by including learning analytics 
research. Here, a comprehensive, systematic review aims to provide information related to EDM’s bibliometrics and 
research trends. Results show that EDM has gained research efforts from scholars from North America, especially 
the United States, as part of a rising publication trend. Of 693 articles that met inclusion requirements, 70% were 
coded as actual data analysis. Statistics, clustering and data visualization were the most popular analytic methods in 
the descriptive analysis, while regression and decision tree were the most popular methods in predictive analysis 
studies. Most EDM research focused on students in higher education rather than teachers or students in K-12. 


1. Introduction 

Various forms of online learning systems have been embraced by institutions and learners, such as 
Learning Management systems (LMSs), Massive Open Online Courses (MOOCs), Intelligent Tutoring systems 
(ITS), and game-based learning systems. In the process of using these online learning systems, huge amounts of 
educational data can be recorded. This information is often referred to as “educational big data”, which cannot be 
analyzed to further provide decision-making support without the assistance of technology. To address this 
technology requirement, researchers have adopted Educational Data Mining (EDM, hereafter) as a solution to 
automate the analysis of educational big data. 

The first International conference on EDM was held in 2008. Since then, more and more researchers started 
to devote research efforts to this area. A widely accepted definition of EDM is considered as, “computerized 
methods and tools for automatically detecting and extracting meaningful patterns and information from large 
collections of data from educational settings” (Kumar & Sharma, 2017, p. 2). The intensity of research efforts were 
also reflected by the number of publications, as nearly 700 journal articles and proceedings were published in the 
period between 2007 and 2017. Among them, 13 review articles were identified (Romero & Ventura, 2007; Baker & 
Kisor, 2009; Romero & Ventura, 2010; Romero & Ventura, 2013; Pefia-Ayala, 2014; Papamitsiou & Economides, 
2014; Algarni, 2016; Salter, Joksimovic, Kovanovic, Baker, & Gasevic, 2016; Shingari, Kumar, & Khetan, 2017; 
Schwendimann et al., 2017; Rodriguez-Triana et al., 2017; Dutt, Ismail, & Herawan, 2017; Muthukrishnan, 
Govindasamy, & Mustapha, 2017). However, several aspects of these reviews could be noted. First, many reviews 
combined articles relating on both EDM and Learning Analytics (LA), which is problematic due to the substantial 
differences between research goals of these areas. Second, two important data sources, the Journal of Educational 
Data Mining and Proceedings of the Educational Data Mining conference, were not included. Third, the review 
sample size and scopes are small (the largest one only encompassed 240 articles from 2010 to the first quarter of 
2013) when compared with the total number of available articles on EDM. Therefore, this review addresses these 
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aspects within existing review articles that cover EDM, then, it outlines a more comprehensive review technique to 
reveal additional findings. 


2. Literature review 

Based on our search, 693 articles and proceedings were published from 2007 to 2017 (major data sources 
will be listed later). Within this number of included articles, 13 review articles were identified published from 2007 
to 2017. The following discussion focuses on the research questions and key findings extracted from these review 
articles. 

There are 7 articles (Romero & Ventura, 2007; Baker & Kisor, 2009; Romero & Ventura, 2010; Romero & 
Ventura, 2013; Algarni, 2016; Salter, Joksimovic, Kovanovic, Baker, & Gasevic, 2016; Shingari, Kumar, & Khetan, 
2017) that did not provide their process for data collection. Five review articles (Romero & Ventura, 2007; Baker & 
Kisor, 2009; Romero & Ventura, 2010; Romero & Ventura, 2013; Algarni, 2016) introduced and reviewed EDM 
methods, data sources and main applications. These 5 reported that EDM methods included not only prediction, 
clustering, relationship mining, distillation of data for human judgment and discovery with models, but also outlier 
detections, text mining and social network analysis. Traditional and computer-based learning environments (such as: 
LMS, ITS, AIH) are all common data collection sources. Although the application of EDM varies, predicting 
student performance and providing feedback for supporting instructors are the two main EDM applications. Salter et 
al. (2016) reviewed 40 tools frequently used for educational data mining for purpose of providing useful information 
for researchers interested in conducting EDM/LA research. Shingari et al. (2017) conducted a survey on the existing 
performance prediction methodologies and found that previous educational records, an excellent educational track 
and learning emotions were significant dimensions for predicting performance. However, the articles were not 
systematic literature review articles. 

The remaining 6 literature reviews (Pefia-Ayala, 2014; Papamitsiou & Economides, 2014; Schwendimann 
et al., 2017; Rodriguez-Triana et al.,2017; Dutt, Ismail, & Herawan, 2017; Muthukrishnan, Govindasamy, & 
Mustapha, 2017) provided systematic review criteria. Two review articles (Pefia-Ayala, 2014; Papamitsiou & 
Economides, 2014) collected related papers to review EDM methods, most prolific researchers, data sources and 
applications. For example, Pefta-Ayala (2014) collected 240 EDM works published during the period from 2010 to 
the first quarter of 2013. The author found that 60% of those works depicted predictive models and 40% offered 
descriptive models; the most typical EDM tasks were classification and clustering; and K-means, expectation 
maximization (EM), J48 and Naive—Bayes were the top-four most deployed algorithms. The most prolific 
researchers such as Ryan Baker, Cristobal Romero and Sebastian Ventura, Kenneth Koedinger, Joseph Beck, etc., 
were also identified. The author also concluded that \ most EDM works were oriented toward student modeling and 
assessment, including performance, behavior, learning and domain knowledge. Papamitsiou and Economides (2014) 
examined the research strategies, learning settings, data sources, analytic methods and research topics and 
categorized them by investigating 40 LA/EDM papers from journals and conferences between 2008 to 2013.They 
found that the majority of articles were exploratory or experimental studies. Virtual Learning Environments (VLEs) 
or Learning Management Systems (LMSs) were most popular learning settings. The authors found that multiple data 
sources, such as system logs, questionnaires, interviews, and open datasets could be gathered for analysis. 
Classification, clustering and regression were the most commonly used analytic methods. The most popular research 
topics include student behavior modeling and performance prediction, the support of students’ and teachers’ 
reflections, and the awareness and the improvement of feedback and assessment services. 

The remaining 4 review articles (Rodriguez-Triana et al., 2017; Schwendimann et al., 2017; 
Muthukrishnan, Govindasamy, & Mustapha, 2017; Dutt, Ismail, & Herawan, 2017) only focused on one aspect of 
EDM applications or the main applications of one EDM method. For example, Rodriguez-Triana et al. (2017) 
reviewed LA and EDM research on monitoring, awareness and reflection in blended learning by analyzing 40 
journal papers from 2010 to 2015. The authors found that these studies mainly focused on formal learning, 
especially in higher education. Virtual learning environments and learning management systems were the main 
sources for data. In addition, action-related and content-related variables were the primary indicators for monitoring, 
awareness and reflection. The evaluation of effectiveness most commonly used a survey instrument to investigate 
perceived usability, usefulness or satisfaction. Only few studies evaluated the impacts on learning with the actual 
performance data. 

Schwendimann et al. (2017) examined educational dashboards by reviewing articles for categories of 
learning contexts, data sources, visualizations and analysis types in both LA and EDM by reviewing 55 journal 
papers between 2010 to 2015. The authors found that most of the learning dashboards were designed for students’ 
self-monitoring and for instructors to monitor students in formal, higher education settings. The dashboards’ data 
sources heavily relied on behavior logs from a single LMS platform. The visualization types, similar to traditional 
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dashboards, utilized bar chart, line graph, table, pie chart, and network graph visualizations. In terms of analysis 
type, the authors revealed that most of the studies in their review were exploratory or proof-of-concept without 
authentic evaluations, and therefore, it was difficult to evaluate the impacts of learning dashboard on learning. 

A total of 59 articles related to students’ performance predictions and published between 2012-2016 were 
reviewed and classified by Muthukrishnan et al. (2017). They found that the annual publications were increasing and 
that the regression model was the most commonly used method for predicting performance. They revealed the three 
major purposes of performance prediction were to predict final course grade, predict performance on the MOOC or 
VLE platforms and to identify struggling students who had potential to dropout. The attributes for performance 
prediction mainly included the student’s personal information, grades and some basic demographics. They also 
reported that only 19% of the articles had considered big data and only 13 articles (22%) used a feature selection 
method. 

Another work conducted by Dutt et al. (2017) provided a systematic literature review on Educational Data 
clustering by analyzing 166 related articles. They found that clustering approach had great advantages in the 
application of student modeling, student profiling and learning style in E-learning contexts. 

In general, different conclusions made by these articles also indicated the scope is too small to obtain 
consistent results. Considering that there are some differences between LA and EDM, this study aims to conduct a 
systematic review on EDM to address these research gaps. 


3. Research Method 

In this research, “educational data mining” was applied as a key term for gathering articles through the 
Web of Science database. Two additional data sources, the Journal of Educational Data Mining and Proceedings of 
the Educational Data Mining conference, were also combined with Web of Science EDM articles to form the raw 
dataset. Because the first related EDM article was published in 2007, the actual resulting search period was from 
2007 to 2017. After the searching stage, 239 journal articles (185 from Web of Science and 54 from Journal of 
Educational Data Mining) and 454 conference papers (693 in total) were collected. Therefore, 693 articles were 
analyzed for the following phase. Then the procedure followed the coding scheme proposed by Du et al. (2018) to 
code each article in order to summarize analysis results. 


4. Results 


4.1 Bibliometrics 
Trends of Publication 

An intuitive distribution of papers from 2007 to 2017 is presented in Figure 1. Only a single paper was 
published in 2007, but 15 proceedings were submitted to the EDM2008 conference. The growth trend has continued 
since 2015 for the number of both journal papers. The whole time span can be divided into two stages, namely, 
smooth growth stage from 2007-2014 and disparate growth since 2015. As the number of researchers started to 
focus on related issues in the field of EDM, more articles have been submitted. 
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Figure 1. The publication trends from 2007 to 2017. 
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Nationalities and authors 

The distribution of the top eight countries and districts for these 693 papers is presented in Figure 2. The 
top three countries are the USA (405), Spain (40) and Canada (33). Scholars from North America, especially the 
United States, are the most active researchers in the area of EDM, followed by ones from Europe. In addition, 
researchers from the USA show increasing research interest dedicated to EDM. 


Total EDM Publications by Country Since 2007 
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Figure 2. The publication trends of top 8 countries. 


First authors’ departments and research areas 

A total of 693 articles were carefully examined to identify the first authors’ departments and research areas. 
Approximately 82% (571/693) of EDM articles have been published in educational journals or at the EDM 
conference. Journals in computer science were also published larger percentages (13.85% (96/693)) of EDM 
articles. Therefore, these two research fields had published 96.24% of the total number of EDM articles. More than 
half of the 693 articles were contributed by researchers from computer science. 


Most prolific journals 

Figure 3 shows the top 10 journals with the highest numbers of EDM publications. First, the results indicate 
the top 10 are educational (60%) and computer (40%) journals. Second, the Journal of EDM, with a total of 54 
articles, far exceeded the number of publications published in other journals, followed by Computer & Education, 
with a total of 13 articles. The journals, including Educational Technology & Society, Expert System with 
Applications, International Journal of Artificial Intelligence in Education, Computers in Human Behavior, 
International Journal of Advanced Computer Science and Applications, Expert System, Internet and Higher 
Education and Computer Applications in Engineering Education, have published 5 to 10 EDM articles from 2007 to 
2017. One notable result is that 116 (49%) out of 239 Journal articles were published by the top 10 journals. The 
remaining publications were distributed in more than 90 journals. Lastly, two journals (Computers & Education and 
Computer Applications in Engineering Education) show an increasing trend as shown in Figure 3. 
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EDM Publication trends by Journal Since 2007 
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Fig. 3. The publication trends of top 10 journals. 
4.2 Research Trends 
Research type 

Four codes, including review, concept or framework only, proof of concept with small scale data analysis 
and actual analysis, were used to reflect the analysis level of these 693 articles. An article coded as a review article 
aims to represent some summarized research findings via literature review, while concept or framework only articles 
focus on the introduction of perspectives, concepts, or frameworks. These two categories do not involve data 
analysis. For example, one smart recommendation system named Micro Learning as a Service (MLaaS) was 
proposed for purpose of delivering personalized open educational resources to satisfy learners’ just-in-time demands 
(Sun et al., 2017). This system was designed based on a top-down approach, and the authors mainly focused on the 
construction of the knowledge base which could provide guidelines for the decision-making process of a 
recommendation system. Because this paper introduced how to construct this system for recommendation without 
data analysis, it was coded as concept or framework only. 

Articles coded as proof of concept with small scale data analysis usually involved the introduction of the 
proposed methods or frameworks, but they validate the proposed methods or frameworks with a small scale data 
analysis. For example, a five-layer framework based on smart computing concept was proposed in order to facilitate 
automated student performance evaluation in engineering institutions (Verma, Sood, & Kalra, 2017). Finally, a 
dataset that included 24 students’ activity data was selected to test the scalability and utility of the proposed system. 
One model was proposed to predict the behaviors of new students for improving the tutoring feedback in an 
intelligent system (Riofrio-Luzcando, Ramirez, & Berrocal-Lobo, 2017). Then, 85 students’ log data was collected 
to validate the proposed model, with the results showing that this model could provide good predictions and 
adaptive tutoring feedback for each student type. 

Articles coded as actual analysis usually reported the steps of data collection, analysis methods, results, and 
interpretations in detail. Implications related to analysis results were also discussed in actual analysis articles. For 
example, 318 students’ dataset that included reading time features, textual features and context features, were 
analyzed based on several supervised classification algorithms to predict if students’ minds were wandering while 
reading (Mills & D’Mello, 2015). Analysis results showed that the mind wandering was negatively related to 
posttest performance which validated the effectiveness of automated mind wandering predictor. Alsheddy and Habib 
(2017) collected 1980 undergraduates’ personal information and academic performance data, including course 
grade, semester’s GPA and cumulative GPA, to build a classifier based on a decision tree algorithm for identifying 
low-performing students at the end of program to support instructors’ decision-making. The authors found that the 
first year GPA was the most important predictor. 

Coding results show that 2% (13/693) of papers were coded as review articles. Approximately 28% 
(197/693) of proceedings and articles were concept or framework only (63 articles, 9%) and proof-of-concept (134 
articles, 19%) with small scale data analysis, and 483 articles (70%) were coded as actual analysis. The results 
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indicate that the majority of EDM research studies involved actual data analysis for discovering meaningful 
information, consistent with the EDM target. 

From 483 data analysis articles identified, the following sections further discuss these 483 articles from the aspects 
of research method and learning environment. 


Research method 

The analysis results of 483 data analysis articles show that only five articles were coded as prescriptive analysis, 
which aims to discover hidden issues and propose the corresponding solutions. Two hundred and thirty-two (48%) 
of the articles, which employed descriptive statistics, data visualization, rules mining or other unsupervised learning 
techniques to conduct their studies, were coded as descriptive studies. Statistics, clustering and data visualization 
were the most popular analytic methods in the descriptive analysis. Two hundred and forty-seven (51%) articles, 
which adopted supervised algorithms, were coded as predictive studies. Regression and decision tree were the most 
popular methods in the predictive analysis studies. 


Learning Environment 

Except for 16 data analysis articles, which did not specify their learning environments nor belong to the examined 
learning environments (Higher Education, K-12 or MOOCs), the remaining 467 articles were examined to identify 
their sample sizes and learning environments. The results showed that (1) more than 95% of these 467 articles 
focused on collecting students’ data for their study, and only 19 articles gathered teachers’ data for analysis; and (2) 
more than 60% of these 467 articles targeted studying issues from within higher education. 


5. Conclusion 

This systematic analysis reveals the development trends of EDM by reviewing related papers from 2011 to 2017. 
Based on the analysis results, several conclusions can be made: (1) the publication numbers of EDM indicate a 
rising trend in the number of journal articles; (2) scholars from North America, especially the United States, have 
devoted the most research effort toward EDM research; (3) researchers from computer science have published more 
than 50% of the EDM articles; (4) the two most prolific journals in EDM are “Journal of Education Data Mining” 
and “Computers & Education”; (5) the majority of EDM research studies involve actual data analysis for 
discovering some meaningful information, which is consistent with the EDM target; (6) statistics, clustering and 
data visualization are the most popular analytic methods in the descriptive analysis, while regression and decision 
tree were the most popular methods in predictive analysis studies; and (7) most EDM research focused on students 
in higher education rather than teachers or students in K-12. 
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Abstract 


Entrepreneurship education has been subject to enormous interest ever since entrepreneurs have 
become the key drivers of business growth across the globe. Entrepreneurial creativity is identified as a core 
competency of large multinational corporations within Thailand, this is because of a unique combination of a 
sense of ownership and innovative thinking. The Thailand 4.0 policy was launched in 2016 by the Thai 
government in order to enhance creativity and technological competitiveness in an effort to move towards a 
creative, innovation-driven economy. However, the 2019 Global Innovation Index stated that Thailand is ranked 
No. 43 of 129 countries in the world, the reason for this being that Thai entrepreneurs and corporate 
entrepreneurs still have a lower level of entrepreneurial creativity, judging from the fact that 90% of products 
developed are not differentiated enough from one another. An in-depth focus group of 8 respondents was 
created using young managers working in a multinational company to understand the triggers and barriers which 
affect the development of entrepreneurial creativity. Additionally, it was found that their entrepreneurial 
creativity is at a moderate level as they are not confident in their own skills driven by current development 
activities, and that any learning experiences are mostly theoretical, one way and passive. They do not feel 
engaged, motivated and confident enough to transfer the knowledge into practice. Design thinking is well 
recognized as a creative, problem-solving process which uses human needs to simulate the journey of 
entrepreneurship, and has been used when developing business education. Social media provides some of the 
most commonly used platforms in daily life, which can incorporate features to allow learning and deliver open, 
collaborative environments. Therefore, learning design derived from a design thinking process is recommended 
to be used in conjunction with social media technology to deliver engaging, experiential, and motivating 
learning experiences among young managers. 


Introduction 


Entrepreneurship and entrepreneurship education have received an enormous amount of interest among 
governments, academic institutions and business industries worldwide. This is due to the fact that entrepreneurs 
are key drivers of business growth, alleviating the slowdown of global economy. Entrepreneurs come into the 
business world harboring unique mindsets and competencies which can include recognizing emerging business 
opportunities, initiating innovative ideas, creating business plans, gathering required resources. They are able to 
do this while maintaining motivation, passion and commitment to achieve goals that come with calculated risks. 
The results have can be seen in developed nations like the United States, United Kingdom, Singapore and Japan 
where entrepreneurs in small and medium sized corporations have contributed at least 50% of gross domestic 
products and have generated more than 65-70% of the whole country’s employment (Yildirim, Trout, & 
Hartzell, 2019). 

In the corporate world, entrepreneurial competency is identified as a core competency in large 
multinational corporations, such as Coca Cola, P&G, Unilever, S.C. Johnson and Sons. It is comprised of the 
ability of executives, managers or employees to take on strong ownership skills, as well as continued innovation 
in their areas of responsibility. This is done by weighing up opportunities and taking calculated risks to achieve 
impressive business results. This is called “Corporate Entrepreneurship” and is used to change the employees’ 
mindsets from a traditional “wait and see”, to a more “proactive and aggressive” method of working (Barringer 
& Ireland, 2012). Skills that should be expected when it comes to entrepreneurship are creativity and business 
ownership. While ownership is more the aspect of engagement and passion across roles and companies, 
creativity is one of the skills that makes up entrepreneurship that can be cultivated within an organization itself. 
Creativity is part of the 4Cs’ learning skill categories of 21“ Century skills. 

Creativity is considered to be one of the most important competencies for success among entrepreneurs 
if they are to become successful in their ventures. Burns and Burns (2014) mentioned the 5 qualities of 
entrepreneurs which are as follows: (1) Creative thinking and innovation (2) Drive for achievement (3) 
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Independency (4) Self-directed/control (5) Awareness of calculated risks. Creativity encompasses multiple 
processes, from recognizing opportunities to creating innovative business models used to start new ventures. 
This is consistent with what was stated by Barringer and Ireland (2012); Bolton and Thompson (2013)and 
Kuratko and Hodgetts (2004), who have included creativity as a key component of entrepreneurs, and can be 
said for creative problem solving, creative thinking, as well as creative design. 

The Thailand 4.0 policy was launched by the Thai government in 2016 in an effort to increase 
creativity and develop a technological advantage that could deliver innovation in the workplace. The reason for 
this being the intention to shift the country from a middle-income and efficiency-driven economy to an 
innovation-driven or creative economy (Murnpho & Unaromlert, 2018). However, the 2019 Global Innovation 
Index (developed by Cornell University, INSEAD and WIPO) indicates that Thailand ranked No. 43 of 129 
countries in the world. Singapore (8"), Korea (11""), Hong Kong (13"), China (14") and Japan (15"") were 
shown to be the best-in-class in Asia. Furthermore, it was found in another study, carried out in 2018 by the 
Organization for Economic Cooperation and Development (OECD), that Thailand ranked No. 3 out of 10 
countries in Southeast Asia, following Singapore and Malaysia in the aspect of innovation and technology 
outputs(The Secretary-General of the OECD, 2018). The opportunity to improve the capability of creativity has 
been outlined here and can be accomplished by tackling the issue of entrepreneurial creativity development. 

Design thinking is well recognized as a creative problem-solving process that takes place though a 
human-centered approach. In itself, the process is not linear, with a strong focus on gaining a deeper 
understanding of users, or customers, who provide a clear problem statement. Potential ideas emerge from being 
able to take on a different point of view, which is then brainstormed and transformed into prototypes to be used 
for testing and retesting, until they fully meet or exceed the customers’ expectations. This process simulates the 
life cycle of entrepreneurs and can foster creative thinking amongst learners (Linton & Klinton, 2019). 

Social media has been used to promote entrepreneurial education via a variety of platforms (Line, 
WeChat, Facebook Messenger), and this has occurred all around the world due unique features provided to drive 
learners’ engagement, experiences and collaborations, and to ultimately improve learning outcomes(Wu & 
Song, 2019). However, there is a limited amount knowledge and studies on the ways social media can be used 
for learning to improve entrepreneurship or entrepreneurial creativity. It was found by Teepapai and Karawek 
(2018) that a company’s culture and learning environment has a strong impact on creative ideas and the 
innovative output of the employees. Therefore, it is recommended to invest in the development of creativity by 
leveraging newly connected technology to promote learning among employees. 

Consequently, this article will explore the literature of entrepreneurial creativity, design thinking and 
social media learning, as well as look at an in-depth focus group study of young managers working in a 
multinational company. Recommendations for a learning design to promote young managers’ entrepreneurial 
creativity are provided at the end of the article. This will take place using design thinking on a social learning 
platform. 


Corporate Entrepreneurship (Intrapreneurship) 

Successful innovation is one of the key success factors for companies looking to achieve business goals 
by growing their revenue, profit margins, market share and corporate image (Miller & Bauer, 2017). Pinchot 
and Pellman (1985) were the first to create “Intrapreneurship” (corporate entrepreneurship) by defining it as an 
individual who has vision above their level of responsibility when it comes to creative ideas. Corporate 
entrepreneurship, or intrapreneurship, is another kind of entrepreneurship which occurs when employees drive 
the business using strong sense of ownership, and will go the extra mile to provide new creative ideas or 
innovations, thus delivering success to the business (Trifan, Guica, & Micu, 2012). Corporate entrepreneurs are 
ones who do not have any intention of creating a venture of their own but are willing to work within an 
organization with a strong sense of ownership. For an entrepreneur to become successful, he or she needs to 
have managers with the corporate entrepreneurship quality if he or she is to deliver innovation(De Lourdes 
Prado, MacHado, Mafra, & Maria Campos, 2012). Corporate entrepreneurship consists of three components; 
proactiveness, creativeness and risk bearing (de Jong et al., 2011) cited in (Miller & Bauer, 2017). Additionally, 
Christensen (2011) cited in (Miller & Bauer, 2017) categorized 4 different types of creative entrepreneurs; (1) 
creative entrepreneurs are the ones that introduce a new venture with differentiated products to capture unmet 
needs (2) creative intrapreneurs are the ones to introduce a new business within a corporation (3) product 
creators are the one who creates new product or services (4) process creators are the ones to continue to improve 
efficiency or create new processes. This makes it evident that there are 3 types of corporate entrepreneurship, 
these include business that takes place within an organization, product innovation and process improvement. 
They are all centered around creating or innovating to increase the organization’s competitive advantage. This is 
why corporate entrepreneurship is identified as a core competency used to generate innovation in the corporate 
world, especially within international firms e.g. Unilever, P&G, Nestle’, and S.C. Johnsons & Sons. It was said 
by Teltumbde (2006) that a corporate entrepreneur or intrapreneur can be more important than an entrepreneur 
as an entrepreneur creates new ventures, while an intrapreneur continues work on an existing business to gain 
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success and achievement. Most often, this is accomplished by adapting and reinventing the wheel based on 
changes in the business environment. 

Menzel (2007) cited in (Hanns C. Menzel, Aaltio, & Ulijn, 2007) explains that the process of corporate 
entrepreneurship has 2 levels; the organizational and individual levels, meaning a corporate entrepreneur can 
occur either as an individual employee or as a group of employees. This procedure commences with opportunity 
recognition and exploitation, then ends with creative solutions or innovations in products, services, technology 
and processes, as shown the figure below. 


Management (organizational level) 


Opportunity New means-ends 
exploitation relationship 


Opportunity 
discovery 


tC‘ 


Corporate entrepreneur (individual level) 


Environment 


Fig. 1. Corporate entrepreneur process(H.C. Menzel, 2007) 


Entrepreneurial Creativity 

Creativity encompasses two dimensions; innovativeness and functional (Runco and Jaeger 2012) cited 
in (Liang et al., 2019). Innovativeness is defined as being new, differentiated, and unexpected, while functional 
includes being useful, handy and practical. Entrepreneurship requires creativity if new business ventures are to 
be made innovative and functional for the market(Sternberg and Lubart,1999) cited in (Liang et al., 2019) 
Entrepreneurship and creativity are interrelated since creativity complements entrepreneurship at every stage, 
from analyzing an opportunity gap in the market, to thinking creatively of potential solutions to address 
customers’ unmet needs, to developing business plans and action plans to deliver an actual product to market. 
Creativity is regarded as a critical factor of entrepreneurship, and is used to identify potential markets, create 
new ideas, and then commercialize them (Saptono et al., 2019). During each stage, uncertainties are sure to 
occur along the way. As such, entrepreneurs or corporate entrepreneurs must employ creative problem-solving 
and innovative thinking to be able to overcome those situations. Entrepreneurial creativity was previously 
defined by Amabile (1997) cited in (Jing & Anja Svetina, 2014) as the creation and implementation of 
innovative business solutions or campaigns, used to launch new products or services. Entrepreneurial creativity 
consists of two key components; creative problem-solving and creative innovation, which are required skills in 
every process of entrepreneurship. Furthermore, entrepreneurial creativity is not regarded as an innate ability, 
but has the potential to be developed from actual, real-life situations. Researchers in the field of industrial 
psychology suggest that nurturing an employee’s creativity is one of the critical success factors which allows 
organizations to gain an advantage over their competition (Ahlin, DrnovSek, & Hisrich, 2014). This is because a 
creative employee is proactive and looks for opportunities to improve their methods of working. In addition, 
they tend to anticipate the consequences of each step they take. Price, Stoica, and Boncella (2013) investigate 
the relationship between innovation, knowledge and performance in family and nonfamily firms, and it was 
found that innovation and creativity drive superior firm performance; therefore, the continuation of developing 
new products, services, processes to achieve business targets is recommended. 

Jing and Anja Svetina (2014) investigated entrepreneurial creativity, and developed the resource-based 
view that entrepreneurial creativity happens during the gathering and allocation of resources, under the context 
of resource constraints. There are 4 types of corporate innovations.(Trifan et al., 2012) 

1. Product or service innovation refers to the launch of new, superior and differentiated products or 

services, when compared to existing products in the market 

2. Process innovation refers to the application of a novel process, used to improve efficiency and 

effectiveness through the introduction of new technology and ways of working 

3. Marketing innovation refers to the launch of a novel mix of marketing components, products or 

services 

4. Organization innovation refers to the application of organization arrangement, re-organization, 

and new-organization structures 
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In a corporation, each department or division can be managed like a venture which requires innovation 
as fuel for success. Therefore, organizations require employees with a strong sense of entrepreneurial creativity. 
Trifan et al. (2012) mentions the 5 senses of corporate creator, explaining the behavior and attitude of 
employees who possess creativity: 

1. Internal Sense describes behavior in which the individual is aware of the firm’s business 
performance, key issues and opportunities. The individual has a strong overall organizational 
network awareness and can accomplish a lot through collaborating with another functional team. 

2. External Sense refers to behavior in which individual has market insight and understands 
competitors so well that they’re able to analyze opportunities and threats which could potentially 
occur in the future. 

3. Positional Sense describes behavior in which the individual can leverage strong a competitive 
advantage for the company, to generate new product ideas to achieve goals. 

4. Strategic Sense refers to behavior in which the individual has a strategic thinking capability to 
evaluate various business decision choices, and to be able to recommend the most appropriate one 
for the company with strong justification. 

5. Value Creation Sense describes behavior in which the individual has a strong sense of ownership, 
and are dedicated to creating value for customers and the company by calculating any risks before 
making recommendations or decisions. 


Internal 
Sense 


Value 
Creation 
Sense 


Positional 
Sense 


Strategic 
Sense 


Fig. 2. 5 Senses of Corporate Innovators (Trifan et al., 2012) 


Design Thinking Process 

Design thinking has been used widely in the fields of business, engineering and architecture to deeply 
understand define, redefine and find the root causes of problems. The purpose of this is to enable the user to find 
the best fitting solutions to said problems through the process of prototyping and testing. In recent years, design 
thinking has come to play a more important role in educational settings. (Koh, Chai, Wong, & Hong, 2015). 
Design thinking refers to a creative problem-solving method that’s systematic and collaborative, and works by 
focusing on a human-centered approach, namely empathizing (understanding), identifying problems, ideating 
possible solutions, prototyping to move from ideas to tangible solutions and testing with target customers 
(Luchs, Swan, & Griffin, 2015). The process itself is iterative and nonlinear, and helps improve the quality of 
solutions provided. This allows for the best match with the expectations of the customer. Design thinking is 
most appropriate for a problem or opportunity that’s not well-recognized, or ill-defined, and this process has 
been used with proven success in the creation of new businesses, the development of new products and the 
improvement of internal processes (Luchs et al., 2015). 

Jaitip promotes design thinking with research-based design, specifically: Digital Learning Design, 
Define Gap, Root cause analysis, Conceptualization, Learning Design, Rapid prototypes, Evaluation and 
Revision (Na-Songkla, 2018). Design thinking originated from the research study of The Hasso Platter Institute 
of Design, or D.school at Stamford University, and refers to the thinking process which allows for varied 
perspective and a deeper understanding of target groups. Via this process, new ideas can be created, prototypes 
and processes can be continuously refined. Additionally, it has been used in conjunction with technology to 
support instructional design to deliver learning outcomes among diverse learners. The design thinking process 
consists of 5 key parts; 

1. Empathize is the first step in understanding a target group in-depth through interviewing, 
observing and collecting evidence of the past successes and failures. This could include 
participation in the target groups’ activities. 

2. Define is the step where data and facts are collected and synthesized actual to identify the root 
cause of a problem. 
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3. Ideate is when people come together in groups to brainstorm different ideas with tools so that they 
can find as many solutions as possible to address the root cause of a problem, which was analyzed 
in a prior step. 

4. Prototype is the step where ideas are transferred so they can become a tangible product or service 
prototype. This, then triggers discussion and criticism from the team and it is essential that this 
step happens quickly if flaws are to be seen and ideas for further improvement are to be developed. 

5. Testing is when the prototype is tested on real customers to observe and evaluate the efficiency of 
a product. The result of these tests is used to continue improving the prototype until it meets 
customers’ expectations. 


PROTOTYPE 


Fig. 3. Design Thinking Process (Stanford d.school) 


Entrepreneurs or corporate entrepreneurs are faced with uncertainties daily, having to cope with many 
ill-defined problems if they are to survive in a competitive market environment; therefore, their lifestyles are 
often very complex and nonlinear. In the past, education for entrepreneurship focused on “about” which means 
teaching mainly in a theoretical manner. This meant that the “through” was neglected, which relates more to a 
direct experience through the journey of entrepreneurship (Linton & Klinton, 2019). The design thinking 
process mirrors the key processes of entrepreneurship and corporate entrepreneurship. For the design thinking 
process, the ultimate goal is to solve problems through the development of creative ideas, products or services. 
Due to this, it is appropriate to encourage the nurturing of entrepreneurial creativity. Linton and Klinton (2019) 
investigated and found that the use of new teaching methods, which incorporated the design thinking process, 
can be an effective approach to entrepreneurial education. The reason for this being that the learners are able to 
undergo more practical experiences as they can experiment in real life situations, and this can help to develop 
both skills and mindsets needed for entrepreneurial creativity. Schiele and Chen (2018) studied design thinking 
and digital marketing skills used in marketing education, and found that learners key skills improved in the 
following areas; understanding, creativity, communication skills, technology skills, critical thinking, and 
collaboration(Glen, Suciu, Baughn, & Anson, 2015). 


Social Media Learning Platform 

It was said by Jaitip Na-Songkla (2018) that the connectivism paradigm emphasizes learning from the 
connection between information technology and groups of people. Social connection is an integral part of 
knowledge creation and the bigger the social connection, the more knowledge is being generated. This paradigm 
has a strong impact on our way of living, communicating and learning. There are 4 facets of connection; social 
media, social networks, cloud-based learning and open education. 

Social media is digital media as a means of communication within social networks using a connection 
to the internet. The users interact with each other by creating their own content as well as consuming content 
created by others on the network, and there are different types of social media that can be used in learning; (Na- 
Songkla, 2018) 

Blogs 

Collaborative Writing 
Video blog 

Photo Sharing 
Podcast 

Virtual Reality 
Crowd Sourcing 


SiO OV ee 
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The benefits of social media learning were outlined by Wu and Song (2019), and are as follows; (1) 
enhance learner engagement, participation and experience (2) integrate outside resources as part of a learning 
environment (3) create a group to allow application practice among learners. Social media enables learners to 
interact with each other, and teachers, in an open and collaborative environment; hence, the rate of participation 
among learners increases. By nature, social media is ubiquitous in all aspects of our lives, in personal, family, 
work and educational areas. Social media is proven to be a key facilitator and driver of creativity (Bhimani, 
Mention, & Barlatier, 2018). It is also used extensively by entrepreneurs for marketing, business networking, 
searching for information, and crowd funding (Olanrewaju, Hossain, & Whiteside, 2019). 

Thailand has the highest rate of social media penetration in Southeast Asia, at 74%, vs. the rest of 
Southeast Asia, at 61%. This translates as 51 million people who are actively using social media, mostly through 
mobile devices. The average time spent on social media per day currently sits at 3 hours 11 minutes. In 
Thailand, the top 5 social media platforms are Facebook, Youtube, Line, Facebook Messenger and Instagram, 
accordingly. The majority 34% of social media users are 25-34 years old.(source : we are social 
https://wearesocial.com) It is evident that social media is an integral part of the professional and personal lives 
of young working adults. Therefore, social media proves to be an appropriate platform where entrepreneurial 
creativity can be promoted among young managers in Thailand. 

Social media as a method for adult and higher education is receiving higher attention in terms of both 
content dissemination and evaluation. It is evident in research results that there are favorable consequences of 
using social media when driving in-depth learning experience and uplifting certain skills, for example: 
collaboration and organization (Stathopoulou, Siamagka, & Christodoulides, 2019). Al-Rahmi, Alias, Othman, 
Marin, and Tur (2018) investigated a model of factors which affect learning performance when using social 
media in throughout higher education in Malaysia. It was found that social media enhances collaborative 
learning and engagement through group assignments. Furthermore, O’ Boyle (2014) studied the mobilization of 
social media in sports management education, focusing on Facebook and Twitter as part of blended learning. A 
conclusion was reached that social media delivered a strong learning platform which has the potential to 
increase student and staff engagement. 


Young managers’ Insights on Development Opportunities 


Focus Group Discussion 

One focus group of 8 young managers in a multinational company were was investigated to understand 
the need gaps of existing entrepreneurial creativity development, so that recommendations could be developed 
in the future. The respondents were between 28 to 35-year-old, with 5-10 years of working experience in 
multinational companies. The discussion guideline covered perceived entrepreneurial creativity skill, past 
development activities and needs for future entrepreneurial creativity development. 


Perceived Entrepreneurial Creativity 

In terms of perceived entrepreneurial creativity, most young managers rated themselves at the 
moderate level and struggled to improve their skill set. The reason for this being that they had become used to 
working under the close supervision and guidance of their superiors and regional teams. In the context of a 
multinational firm, there are always standard operating procedures (SOP) as guidelines for the ways of 
completing tasks and even templates to be followed for each piece of work. This is because global and regional 
teams will need to consolidate and compare different countries and regions, meaning that it is not allowed for 
workers to deviate from the template or structure. If young managers are asked to conduct a piece of work 
without the use of a template, they would find it difficult as they would not know how to start. This is called 
“Template Syndrome”. Most of the staff members came from sales and marketing teams and so they need to 
create a marketing plan on a yearly basis. However, they receive templates from their global and regional teams 
to follow rather than beginning from scratch. In the end, they would fill in forms rather than thinking creatively 
and analytically about the strategy that should be proposed. 


Past Entrepreneurial Creativity Development 

Entrepreneurial creativity has been integrated as part of the managers’ functional competency for sales 
and marketing. They are all evaluated on this in their performance reviews; however, they are not clear on 
exactly what it means to their role and haven’t had a chance to practice in real-life scenarios. Most of the past 
development focused on theoretical framework in an e-learning module, rather than real-life experiential 
learning. This means that there’s a limited amount of collaborative working sessions with other learners. They 
are required to complete a session by a deadline. Yet, they do not feel engaged in the materials because they 
mostly spend time alone on e-learning modules, being tracked by their human resources or training teams. 
Training feels more mandatory to them, and does not inspire them to develop themselves. 
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Needs for Future Development 

In terms of wish lists for future entrepreneurial creativity development, the managers are asking for 
higher quality development sessions by focusing on real-life case studies. This would allow them to work with 
peers and learn how to transfer knowledge to practice. Nevertheless, they are not against e-learning, though they 
do think there should be mixture of e-learning and in-classroom activities where they have the opportunity ask 
questions and engage in discussion with trainers and peers. They are expecting a workshop session where they 
can use personal experience from their own work in discussion with other learners and trainers in the workshop 
session. The e-learning platform shouldn’t be restricted to one-way learning and should be accessible anywhere, 
any time as currently it can only be accessed via laptops using a VPN, and is not mobile friendly. 

In conclusion, there is a lot of potential to revamp entrepreneurial creativity development through the 
enhancement of experiential learning, as well as boosting engagement among learners. Learning design should 
stimulate deep and collaborative learning in real working situations; thus, providing opportunities to reflect on 
current work and ways to improve upon the completion of daily tasks. Engagement, experiential activities, 
collaboration and inspiration are the keys to success in this regard. 


In conclusion 


Promoting young managers’ entrepreneurial creativity using social learning platforms 

From the literature and results of focus group discussions, I’d like to propose a learning design for 
entrepreneurial creativity by utilizing the design thinking process in order to mimic the experiences of corporate 
entrepreneurship. Empathizing, defining, ideating, creating prototypes and testing would all be incorporated into 
new product development. Following is the proposed learning design in combination with design thinking 
processes and cognitive tools. Technology is another tool to be used in this proposal, and allows for support of 
experiential, collaborative, real-life based learning design. The content of the case study should be based around 
the category of the product that the company developing, or an expansion plan for future development. This will 
allow the team have a basic understanding of the field of work, and have target customers’ insights available to 
them. 


Table 1. depicts the proposed learning design for entrepreneurial creativity 


Design Thinking Process — Cognitive Tools Enabling Technology 
Learning Design 
1. Empathize: - Interview & Observe guide Purpose: Interact with target 
- Interview and observe target - Customer Persona customers 
groups to understand the customer | - User profile canvas - Line Video Call 
journey and pain points for - Facebook Messenger 
assigned product category 
2. Define: - Fishbone Purpose: Collectively define 
- Define the key unmet needs of - WH questions (What, When, customers pain points 
target customers and root causes Where, Why How?) - Line Notes 
of these needs - Line Albums 
3. Ideate: - Brainstorming sessions Purpose: Collaborative platform to 
- Brainstorm with team on as - Idea communication sheet gather ideas and discuss 
many solutions or product ideas as | - Creativity techniques - Line Album and Notes 
possible 
4. Prototype: - Product Concept board Purpose: Platform to display ideas 
- Create product concepts and - Boxing and shelfing and prototypes 
initial prototypes - Line Album and Notes 
5. Test: - Conduct A/B testing Purpose: Interactive forum to test 
- Conduct a focus group interview | - Focus group discussion ideas on target customers 
to test on target customers and - Use feedback-capture grid - Line Album 
gain feedback which then allows - Line Messaging with groups of 
for improvement potential customers 
- Line Voting 
- Line Survey & Feedback 


This is the initial outline of a potential learning design and will need to be rigorously evaluated to 
validate what will work for young managers, and to enhance the effectiveness of entrepreneurial creativity 
development. For future research, it is recommended to: 

1. Conduct a survey on a larger scale of needs assessments of entrepreneurial creativity among young 

mangers. 


479 


2. Research and design the learning design, components, and technology which are to be used when 
developing entrepreneurial creativity. 

3. Test and validate learning design components and technology among young managers to propose 
the most appropriate model and platform to be used in learning design. 
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Abstract 


In the 21" century, teachers need to continuously develop their knowledge and skills to work creatively 
on new ideas. ISTE (2008) standards outlined the teacher’s model, which has emphasized on the improving of 
learning and enriching professional practice by using a variety of digital age media formats. Similarly, The 
Office of the Basic Education Commission of Thailand (2010) defined teacher qualification competency as both 
a core competency and a functional competency. A pre-service teacher who is likely to pursue a career in the 
education system needs to develop these competencies during their period of study. Core competencies such as 
a working motivation for achievement, self-development, and teamwork are required for a pre-service teacher in 
a teaching preparation program. As for functional competencies, curriculum and learning management is listed 
as the most important part of professional development. When the goal of instructional design practice has been 
set, Self-Regulation Learning is a crucial tool used to control responsibility of one’s learning, and consists of 
self-generated, reflective, and strategic engagement in academic tasks (Zimmerman, 2002). However, from a 
social-cognitive perspective, the causal factors outlined for motivation include environmental factors. Due to 
this, self-regulation is a multifaceted process in thought and behavior (Bandura, 1991). During the period that 
pre-service teachers first experience a real classroom, social support from supervisors in school, university 
faculty or even their peers can be considered to be environmental structuring in which the learner engages in a 
common tasks in order to solve problems, complete tasks, or create products. Note-taking is one of the strategies 
which serves as a cognitive tool and allows individuals to become more critical in the reflective cycle. However, 
this study proposed a framework for an online collaborative note-taking strategy which creates benefits derived 
from reflection in both learning contexts and the collaborative learning. The aim of this being to help pre-service 
teachers adjust their personal standards and reach standardized norms during supervised, field-based teaching 
experiences. The collaborative engagement process was designed to have three phases: forethought, 
performance control, and self-reflection. All of these are used to promote self-regulated learning when pursuing 
a classroom management based goal in the instructional design practice. The first phase is comprised of the 
analysis of practical problems by researchers and practitioners (Christensen & West, 2018). 


Introduction 

Pre-service teachers begin the process of developing instructional design in schools while engaging in 
practice for later professional experiences. The practices require a self-regulated process to perform 
successfully, from planning the instructional learning design, to controlling their performance in learning design 
and teaching, to becoming a reflective teacher in order to improve the effectiveness of their teaching. The 
process of coordinating work from supervisors in schools, university faculties and among their peers all 
provides feedback and supports collaborative work. Therefore, the development of a self-regulatory process in 
instructional design practice is extremely important. Consequently, it is stated individual motivation may arise 
from environmental factors for regulated, pre-service teachers working and learning within this context. This is 
especially so when they are working collaboratively in learning design and development, and corresponds with 
the theory of social cognitive learning, Bembenutty (2011). 

One problem that can be observed is that pre-service teachers’ professional experience gained from 
working in real-life environments often lacks effective self-regulation in learning design and development. 
Because they are teaching for the first time in a real-life setting, pre-service teachers must deal with various 
scenarios, arising both from the pressure of the institution and the school. Although adaptation in every area is 
important when becoming a professional teacher, instructional design remains an important issue to consider in 
the early stages of teaching. In some cases, students may focus on other aspects of adaptation and forget that 
they need to develop the effectiveness of their teaching. As Sever (2016) stated, instruction refers to the starting, 
leading, easing and implementing of learning and effective classroom management. In this research, we defined 
self-regulation as the process of designing classroom management through three important steps, namely: the 
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forethought, performance, and reflection phases (Zimmerman, Bonner, & Kovach, 2009). This was completed 
using collaborative notetaking to increase the efficiency of effective instruction. 


Research Design 

Design-based research is a systematic approach used to plan and implement innovations by working 
collaboratively with practitioners over three phases, including: analysis and exploration, design and 
construction, evaluation and reflection (Huang, Spector, & Yang, 2019). This paper presents the first phase of 
the study “analysis and exploration’, as well as “the early step of design and construction”’. In addition, design 
thinking was introduced in conjunction with the utilization of knowledge and practice to find viable solutions 
and meet needs, primarily using a technology-driven approach (Koh, Chai, Wong, & Hong, 2015b). We applied 
the design thinking approach to be able to bring valuable insights into the practice of design-based research. As 
a result, design thinking will play a multidisciplinary, collaborative role to produce innovative products, 
systems, and services (Christensen & West, 2018; Koh, Chai, Wong, & Hong, 2015a; Skogstad & Leifer, 2011). 


The elements and outcomes of this early stage of research are shown below in table 1. 


Table 1. The elements and outcomes from the early stages of design-based research adapted from 
(Huang et al., 2019) 


Design-based research 
phases 


Design Thinking stages 


Elements 


Outcomes 


Phase1: Analysis and Empathize Learn from an audience _ User needs and 
exploration via a semi-structured understanding of 
interview problems 
Define State the problem and Statement of problems 
role of objectives 
Early steps of Ideate Review of literature Learning design 
Phase 2: Design and Draft principal to guide _ blueprint 
construction the design of blueprint 
Propose framework 
Prototype Prototype 


Throughout the empathize stage, we collected qualitative data using a semi-structured interview with 
two pre-service teachers and one supervisor in a school setting. This was done to identify real, problematic 
conditions, the result is summarized in the following table. 

Table 2 Qualitative data from the empathize stage 


1“ Pre-service teacher 


2™ Pre-service teacher 


English Teacher 


from Advanced English from Computer and (Context of supervisor in 
major. special education major. _ school setting). 

Task Analysis Consider the issue of Looking for what kind of No fixed criteria set for 
classroom management, _ learning environments students and do not force 
therefore determine occur in the classroom them. Let students 
learning environments and then think of the analyze content and 
that are appropriate for issues that are organize time- 
the content and time. appropriate for that management by 

context. themselves. 

Planning Using one style of Start with the students’ Have students make a 
classroom management and the school’s needs. draft of classroom 
for the whole semester. management including 
However, it has been both the learning 
found that the planning environments and 
was not as good as it classroom rules. 
should be. 

Pattern of Working No fixed format but must _ If it is found that Have students propose 
submit a plan to the learning activities don’t _ the classroom 
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teacher a week before 
presenting it to the 
mentor. It took 2-3 hours 
to develop a classroom 
management plan and 
they are completed at 
one time. 


work, then cancel the old 
ones and start to revise. 
One plan can be 
completed within 3-4 
hours. 


management plan, but 
without changing their 
line of thought. Just 
suggest situations which 
can occur when using 
this activity. 


Pattern of Notetaking 


Note-taking for ideas by 
writing important topics 
in order so as not to 
forget about them, 
sometimes putting the 
bullet and content on the 
presentation slide. 


Note in diary or note on 
the phone as bullets and 
checklists of media items 
to prevent forgetting. 
Sometimes interesting 
content will be recorded 
as hyperlinks. 


Encounter Face-to-Face 
comments because when 
sending via mail or 
sharing a folder, teacher 
doesn't have enough time 
to check, teacher prefer 
to sit and watch and fix. 


Help-Seeking 


Start thinking mostly on 
their own at first, 
sometimes asking friends 
if the activity is possible 
or not. Most of them 
consulted friends who 
went to the same school 
in an informal 
environment, like while 
they were having lunch. 


Rarely ask for help from 
others. Like to work 
individually at first and 
then ask friends for 
input. 

Seeking help from 
friends when meeting 
face-to-face, for 
example, at seminars. 


Providing help only 
when students face real 
problems but will not 
block their creativity 
when developing ideas 
or organizing activities. 


Help-Seeking Channel 


Verbally discuss with 
friends on the Line 
application. 

Send an email to their 
supervisor, if the 
professor has time, they 
will join the student to 
aid in solving their 
problems. 


Like a face-to-face style. 
Like to do it themselves 
at first, then asking 
friends who teach at the 
same level for comment 
sharing. 


Working Process 


When they do not have 
experience in a certain 
area, they tend to ask for 
opinions from people 
who know how the 
subject will be taught. In 
reality, the supervisor 
has high workload, 
meaning that they are not 
always able to help. 


Hand in classroom 
management plans via 
email, if there is time, 
the teacher will make an 
appointment to discuss it 
with the student or if 
they don't have time, 
send comments via 
email. 


For the first comments 
that are provided, it takes 
3-4 rounds of returning 
feedback. After students 
have a greater 
understanding, then 
frequency will be 
decreased. 


Pressure Serious mentors Pressure occurs with - 
sometimes do not give high self-expectations 
positive feedback, even and not understanding 
if it should be. Giving the content that needs to 
negative comments can be taught. 
cause work pressure. 

Self-Control Supervisors have high Reinforce and reward, - 


expectations to do well. 
They must also think of 
students who have 


this could be done by 
buying things for 
oneself. 
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expectations of pre- 
teachers. 


Forcing oneself to finish 
tasks. 


Learning Motivation 


Looking at the benefits 
students receive is 
important. 


Watch programs that 
inspire and teacher with 
whom you have a good 
relationship to find a 
solution to the problem. 


Time-Management 


Have a time limit with 
the scope of work being 
made clear, then choose 
to manage time by 
oneself. 


Not doing as well as one 
should, due to not 
working gradually. Non- 
completed work will 
become a burden in the 
future. 


Strategic Planning 


Prefer the style of sitting 
and talking the most. 


Work what is preferred 
personally at first, then 
gradually complete other 
tasks. If it is an 
important job, do it alone 
to concentrate and 
relieve the pressure from 
people around. 


Provide feedback before 
and after the lesson. 
However, not 
recommended by the 
teacher as it will block 
creativity. 


Self-Reflection Reflect upon comments —_ Write it as a short note - 
from a mentor often to on a Post-It and then 
improve andimplement write it again later, in a 
changes. Not often post-teaching record 
talking to others after document. 
teaching because the Do not use the notes 
focus ison making new _ after teaching for 
learning management discussion but keep 
plans. records for discussion 
with supervisors. 
Platforms Use email because the Have conversations and _‘ The teacher has a lot of 
file doesn't disappear. talk face to face due to work and is not always 
fear of misinterpretation. able to use email. 
Therefore, choose to use 
an offline channel or 
allow the students to take 
the computer with them 
to edit on the computer 
immediately. 
Collaborative Useful but may not The concept of watching Able to work together, 
Notetaking always have to be done together because ithelp both when typing and 
because it will increase other to understand as writing. 


workloads. 


well. 


Defining change of problems and assumptions 

From the semi-structured interview used during the empathize stage, the true root of the problem was 
found. During the forethought period, most pre-service teachers had the same method of job analysis. When 
planning, long term thinking was usually focused upon for classroom management. As for the work style, it was 
found that there was no exact pattern. Most of them tended to design classroom management to be based around 
the whole semester, which takes an average of 2-4 hours. While supervisors prefer to see intensive work during 
the first period of a teacher’s professional experience, pre-service teachers usually relax in terms of strictness 
after they have developed a good grasp of the guidelines. However, this must not prevent the creativity. 
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Regarding strategy, it has been found that students prefer to sit and talk most, and they will also do work that 
they want to do first, then gradually move onto other pieces of work. As important work requires a lot of 
concentration, it usually is usually done alone, relieving pressure placed on the worker by people around them. 
Students often want to be alone when spending time designing learning environments. Therefore, they would 
like to be provided with comments before the design process, and then to receive feedback after the teaching is 
finished. 

During the design process, students often record ideas in a small notebook or on mobile devices, and 
document only the important topics. In the note-taking process, they select links that they see necessary to put in 
a personal note. Students often draft content into a presentation slide, then adding said content to the learning 
management plan later. Despite this, it is preferred by the supervisor of the educational institution to engage in 
face-to-face feedback, for the reason that they do not have enough time reply to emails. In addition, help- 
seeking personalities mostly tend to plan alone. If unable to complete tasks, they will ask for the opinions of 
friends, for example, whether it is possible to complete the activity or not. Most of them would consult friends 
who went for teacher training at the same school in informal environments, such as while eating lunch. The 
main method of asking for help consists of verbally communicating with friends, using the LINE application 
and email as communication tools. However, in a real-life situation, it was found that most students still had to 
print out the learning management plan, and then send it as a document to the teacher. The plan was then used 
by supervisors to sit down with the student and engage in further discussion. 

It was found that most pre-service teachers did not like it when the teacher gave direct instructions on 
what to do. They tended to complete tasks firstly by themselves, then taking on comments later. While thinking 
about design, classroom management plans are often shared with peers who are studying on the same level as 
themselves. They tend to ask for opinions only from those who are not familiar with the scenario at hand. Due 
to the supervisor having a high workload, they were often unable to help at all the time. In terms of work 
pressure, pre-service teachers like to receive positive feedback to increase positive motivation. In addition, 
rewarding oneself is another method that can be used to motivate. Sometimes, they choose to use media such as 
inspiring videos that speak about topics which motivate them in order to design an effective classroom 
management plan. As for time management, pre-service teachers like the time and the scope of work to be set 
out clearly, so that they can manage their time by themselves. However, it was found that most students do not 
manage time well. 

During the stage of reflection, it was found that students lean towards writing their own short notes on 
post-it notes. Supervisors would provide a lot of comments to be used for improvement and then write 
summaries after teaching in their own format. However, most did not talk to others after teaching because they 
were inclined to focus on making new learning management plans. Lastly, when it comes to the platform that is 
used, there are students who use email because the file can be kept as evidence and does not end up being lost. 
Though, they also like face-to-face conversation, because they fear misinterpretation and they can ask about 
items that they do not understand clearly. Conversely, the supervisors have a very high workload and are not 
always able to use email. Therefore, they choose to use communication methods such as the LINE application, 
or turning on the computer and sending feedback immediately. Lastly, if there were a platform that allows them 
the opportunity to take notes together, it would be useful, but they may not have time to use it after each session 
because of their heavy workloads. 


Additional findings 
The assumptions that were made are both consistent and inconsistent with the qualitative data. In 
addition, it was found that if the framework was developed to facilitate the self-regulation of learning in the 
design of classroom management, it should have the following important characteristics; 
e Set aclear work goal. 
Specify the duration and scope of work. 
Customize notification styles, manually configuring notifications. 
Chat both synchronously and asynchronously. 
Formulate one's own work strategy. 
Save ideas in personal notes in an unofficial format. 
Supporting and seek help when needed. 
Create a submission area and keep notebooks together. 
Reward and send the positive feedback in various forms. 
Allow the media to create motivation. 
Build a collection of evidence and trace work. 
Self-control tools. 
Self-reflection tools. 
Self-assessment both in the form of checklists and ratings. 
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Ideate Stage 

The creation of solutions to problems must use previous theory, as well as research and practice which 
consist of the synthesis of theories related to instructional design in this research context. Self-regulated 
learning, collaborative note-taking and communication platforms are also important in this process. This allows 
us to develop the features necessary to create the prototype as follows; 


Pre-Service Teacher Instructional Design Practice 

The demand to help students develop 21st century competencies is increasing, so we need to reevaluate 
prospective teachers who are developing their potential to work creatively with ideas for sustained teaching 
improvement (Koh et al., 2015b). As stated in the ISTE Standards (ISTE, 2008), the effective teacher model is a 
teacher who inspires student learning and creativity using their knowledge of subject matter, teaching and 
learning, and technology. These teachers will also able to communicate relevant information and ideas 
effectively to students, parents, and peers using a variety of digital age media and formats. Consistent to 
UNESCO (2008), and according to the ICT teacher Competency Standards, the professional development of 
teachers is a key part of becoming master learners and knowledge producers themselves when in teaching 
practice. Effective teaching and learning cannot take place without applying classroom management 
sufficiently. Therefore, a teacher will take the role of classroom manager, setting the classroom rules and 
creating an effective learning environment (Sever, 2016). Davis (2018) proposed a solution of bringing in 
stakeholders including university faculties, supervisors, cooperating teachers, and pre-service teachers, and 
having them work together to incorporate and agree upon a positive learning environment to be used in the 
classroom. Lawrence-Wilkes and Ashmore (2014) stated the role of reflection in learning and development, as 
embedded in professional education for educational practitioners. This allows for continued professional 
development, work-based learning and lifelong learning. Teachers might reflect on the contextualization of 
multiple factors in order to create both a layout and the students’ preferred learning environment (Hamilton, 
2019). 


Self-Regulated Learning 

When a goal of instructional design practice has been set, self-regulated learning become a crucial 
process used to control responsibility. Self-regulated learning consists of three keywords: Learning, which 
means activities in the pursuit of knowledge and development of governance skills. Regulation, which is the 
examination of work in its current state compared to the goals that have been set, such as learning goals. Self, 
which means the context of achieving one's personal goals. Therefore, self-regulated learning means the 
acquisition of knowledge and skills done in a self-navigated manner, and using one’s own motivations. Students 
can set their own goals and learning strategies in order to achieve main goals by relying on periodic self- 
assessment, comparing current work and strategies to goals set, and then adjusting strategies as needed (Hall & 
Goetz, 2013). Whereas Zimmerman (1989) proposed the cycle of regulation including person, behavior, and 
environment which all affect each other, and later on, Zimmerman (2002) specified the model of self-regulated 
learning consisting of 3 important phases, which are the Forethought Phase, Performance Phase, and Self- 
Reflection Phase. 


Collaborative Note-taking strategy 

Previous research has shown different methods of notetaking. E.g. The Learning Strategies Center 
(2001) Cornell University has outlined a process of taking notes, including recording lectures and using note 
columns. Other methods include the use of questions to reflect on the content by asking oneself questions. An 
example of this would be, what is the importance of facts? What are the principles? The final method is a review 
that takes at least ten minutes every week to check what has been recorded. This process can be applied to the 
design of teaching and learning conducted by students in the teaching profession. However, from a social 
perspective, Seel, Lehmann, Blumschein, and Podolskiy (2017) have revealed the three presences (Social, 
Cognitive, Teaching) and their relation to instructional activities. This concept is in line with the social cognitive 
theory that was originally developed by Bandura (1976), who described social learning theory as a learning 
process that can occur from other people. This happens through the observation or modeling of the experiences 
of others, without the need to experience trial and error by oneself. The learning process occurs during 
interaction with an environment and taking part in an observing or imitating process. 

We brought both concepts together and integrated them with emerging technology which has resulted 
in more notes being made online. Yang and Lin (2015) proposed online collaborative notetaking strategies that 
allow other people to use online memo boards, which can be combined on the same sheet for the same period. 
Reflection was defined by Glasswell and Ryan (2017) as a collaborative process which engages others to enable 
them to think differently. This process can be applied to teaching design in both learning planning and 
controlling of efficiency in learning design. Pre-service teachers might reflect in order to improve efficiency, 
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encourage stakeholders to participate, expand ideas, and provide necessary support and feedback. In this study 
we synthesized the tools and platforms that can be used by pre-service teachers for collaborative notetaking. 


Table 2. Synthesis of the right tools for collaborative notetaking 


Tools Goodnote Noteability Evernote Word Onenote Google Sketchboard Microsoft 
Document Sheet Whiteboard 
Onedrive 

Web Application x x x v v v v v 

Mobile Application v v v v v v x v 

Import files v v v v v v x x 

Typing v v v v v v x x 

Collaborative-typing * x v x v v x x 

Free Handwritten v v v v x x v v 

Drawing Pen x x v x x x v Vv 

Sharing to others v v v v v v v v 

Free x x v v v v v v 

In App-purchases v v v v x x v v 


A Proposed Framework 

The design of the online collaborative note-taking strategy used in Self-Regulated learning for pre- 
Service teachers is still in its first stage. Zimmerman's (2002) Self-Regulation model was adopted by the 
researchers, consisting of three main steps, namely the Forethought Phase, Performance Phase and Self- 
Reflection phase. This was done by identifying the sub-steps of issuing self-regulated learning at each stage that 
is necessary and then dividing into the categories of teaching and learning under the roles of the learner and 
instructor. We have drafted the concept of collaborative notetaking to be used as a working tool. Then, the study 
of related concepts and theories allowed us to develop the prototype on an online learning design platform 
together with the instructional design process. 


Table 3. A proposed of learning design blueprint 


Steps Activity Instructional Roles of Learner Roles of 
media/ Technology Instructor 
Forethought Set clear goals from Collaborative Sheet Set the terminal - 
Phase designing a long Task lists goal 
lesson plan 
Determine the Online Calendar Create a working Share a working 
duration and scope timeline timeline with 
of work Share the timeline instructors 
with instructors 
Customize the Reminder Set notifications - 
notification form Notification 
manually 
Meet for a chat both Chat Communicate with Provide feedback 
synchronously and Link to Social others as necessary —_as necessary 
asynchronously Media Platform Senda 
reinforcement 
message 
Define their own Task lists Set working - 
work strategies strategy and tasks 
Performance Create a work Online Notetaking Create Online - 
Phase delivery area and Collaborative 
Create Online Notetaking tools 
Collaborative 
Notetaking tools 
Collect ideas in an Online Notetaking Taking note online = - 
informal format via (Free style format) 
personal notes 
Support and seek Collaborative Seeking help or Providing 
help when needed Notetaking (sharing Giving help support as 
setting) requested 
Chat 
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Link to Social 


Media Platform 
Provide rewards and Digital Badges - - 
positive feedback in Hyperlink 


different ways 


Provide various 


Video, Hyperlink 


Post links or 


types media to videos 
create motivation 
Collect learning Personal Drive Collecting work - 
evidence Submit Task button 

Self-Reflection Self-control and Progress bar Reflect - 

Phase self-reflection Checklist 
Reflect yourself Forms Self-Evaluation Student’s 
with use of Portfolio Evaluation 
checklists and 
ratings 


+ New File 


Notebook 


Your Notebook 


60 To NoTEBOOK — 


T Notification Alert: 


Progress: 


Lesson Plan Status: In_ Reflection 


Development Utilize 


Band Hold 


Queue Ideate 


Reflection 


Fig 1. The wireframe and prototype developed for collaborative notetaking 


Future Design direction 


With the use of the first prototype, we have found weaknesses in the prototype which are; 


- The researcher was unable to develop collaborative notetaking at this time. Therefore, links from free 


to use, external readily-made re 


- From the study, it was discovered that there is no application that has the ability to record both typed 
and free hand notes together, including the ability share and giving further opportunities to 


collaborate. 


Further Research Work 


This paper has thrown up the ideas of a conceptual framework of online collaborative note-taking strategy 
in self-regulation learning for pre-service teacher, therefore, this is a guideline for furthering and creating a 
learning community and promote productivity to improve classroom management and can be applied in the 
professional development process to achieve the competencies necessary for future professional practice. 
However, this is in the early stage of research, further work needs to be done to establish the fully develop web 
application and whether to investigate the ease of use and the effects of using the tools on level of self-regulation. 


sources are required. 
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